What Is AI Development? A Practical Guide to How AI Products Actually Get Built

AI development is creating software systems that use data to create forecasts, develop new content or help people make decisions. This includes everything from defining the issue you are trying to solve to placing the completed product into operation and tracking how well it performs. The vast majority of the effort in developing an AI application is simply normal software engineering surrounding a model; not the model itself.

AI development refers to the steps required to design, develop, test, deploy, and maintain an application developed using machine learning algorithms. In contrast to traditional software, the behavior of an AI application is based upon the patterns it learns from its training data, rather than being explicitly defined by a developer's code.

In practical terms, it is when a single team develops a data pipeline; develops a model; develops an evaluation set; and deploys to production. For instance, a fraud detection system, a customer support chat-bot powered by document search, and a demand forecasting application would be examples of systems developed using the same processes - developing high quality data; selecting a appropriate model; conducting thorough testing; and continuously monitoring performance.

Traditional software you always get the same output from the same input. With AI systems the output varies and can be incorrect in ways that were never programmed into it. This means that testing, shipping and maintaining your AI system will require a different approach.

In contrast to conventional testing where each test is evaluated based on whether it passes or fails; when using AI, testing is evaluated based on its statistical performance against an established set of criteria (evaluation sets) and with input from humans. The performance of an AI system may degrade post-launch as the data used in the real world evolves, therefore, the process of monitoring should be continuous. From the beginning, budget and create a strategy to accommodate the continuous nature of monitoring.

The use of the name artificial intelligence (AI) is the overarching term; machine learning is a subset of AI, which learns through the use of data; and, deep learning is a subset of machine learning, which utilizes multi-layered neural networks. The large language models (LLMs) are examples of deep learning models.

This matters because "AI development" can mean many things. A simple regression model to predict churn, or building an app with a large language model via its API are both examples of AI development. Since there are so many different types of AI development, each having a different amount of cost and difficulty, deciding what level is best for you will be one of the first decisions you make.

The AI development process consists of seven stages. These are framing the problem; preparing data; choosing a model approach; training/prompting the model; evaluating the model; deploying the model; and monitoring the model. It is rare for an AI project to be successful if it skips one of these stages.

• Problem framing: Define the Business Outcome and How Success Will Be Measured Prior to Building any Models.
• Data preparation: Data collection, data cleaning, and data labeling. If the data is of low quality then no matter how good the model is, it will produce poor results.
• Model selection: I'll help you select the most suitable model. You have several choices: use an existing API; create a new model by building upon retrieval; fine-tune an existing model; or develop a completely new model from the ground up.
• Training or prompting: The training/prompting option allows for either training a model to perform the required action, or creating prompts/retrieval systems that will enable an already trained model to perform the required task.
• Evaluation: The evaluation process involves testing your system with a sample that is representative of the whole, and then reviewing each failure individually to determine what went wrong.
• Deployment: Create a way to use the model through APIs, provide security, and make sure the model works with your existing product. Good DevOps and reliable infrastructure will help you serve the model reliably.
• Monitoring: Monitor your performance (accuracy, latency, cost, and drift) to see if you need to retrain or adjust your model when those metrics begin to decline.

The Hadrus Build Ladder provides a four-step model for determining what level of artificial intelligence (AI) engineering a problem requires. It is suggested to only advance through the model as far as the problem itself dictates. As you move up the ladder, each addition to the process will increase costs, time, and risks.

• Rung 1: An off-the-shelf AI service available through an API. It is the fastest and least expensive method of using AI. It can be used for common tasks such as summarizing text, categorizing data, and converting spoken words into written text.
• Rung 2: Systems with prompting and retrieval augmentation. This system uses the user's documents/data to provide answers to the questions asked by the system. This is a great system for providing helpdesk type assistance as well as internal knowledge management systems.
• Rung 3: You use an adaptation of the model to fit your domain, style/voice and/or format requirements when using only the prompt is not enough.
• Rung 4: Custom-trained models. These models are created based on the specific data you provide to perform a highly targeted predictive function. However, this level has the highest associated cost and takes the most amount of time to develop.

Many teams try to go straight to Rung 4, which is the most advanced level. Our experience shows that most business issues can be resolved at either Rung 1 or Rung 2; and the funds saved from resolving them will provide the resources needed to evaluate the effective delivery methods to protect quality. To find out more about how we deliver results, visit our How We Work page.

An AI system used in production will have 6 parts. Those are: Data Layer Model Layer API Layer Orchestration Layer Evaluation Layer Observability Layer The model is merely a part of that whole system.

• Data Layer: contains all data used for training/hosting models.
• Model Layer: contains the model(s) that have been trained/hosted.
• API Layer: exposes models to applications/products.
• Orchestration Layer: controls the execution of multi-step logical processes (i.e. retrieving documents prior to responding).
• Evaluation Layer: continually monitors the quality of the model's output.
• Observability Layer: tracks costs associated with running models, latency of model response times, and error rates of model outputs.

It has been found that many organizations who focus solely on investing in the model layer find themselves needing to address the additional 5 components at a later date. This represents what we call the "unglamorous plumbing" of our AI infrastructure/MLOps efforts.

Many businesses use one of two main types of architectures to route requests from the user, through their application, then an orchestration layer, to a model, and finally return to the user; all while using other data stores and monitoring systems throughout the process.

• Retrieval-augmented assistant: Users ask questions. Your system looks through your documents in a vector database, then sends the appropriate sections to a language model so it can generate an answer using your content, and provides references for the information provided. Using this method will allow you to keep the generated answers as close to what is found within your content as possible, making it easier to review.
• Predictive model service: New records are sent to your application; you receive back a score, i.e., churn risk, demand forecast, etc. The model was trained using historical data and is accessible through an API. Retraining the model will occur on a regular schedule as new data becomes available.

AI development services are organized into 4 categories: custom AI development, AI software development, AI engineering and product-focused/LLM-focused builds. The type of service required will depend upon where your company is in its AI journey - exploring, developing a feature, or operating AI at scale.

The creation of AI-based features (e.g. agents, copilots, RAG pipelines) is achieved through custom AI development which creates solutions specifically designed around your data and workflow. Integration of AI features into applications can be accomplished through AI software development. AI engineering provides the necessary infrastructure, pipelines and operations to ensure that your system remains reliable. Product and LLM builds enable the conversion of a concept into a usable, shipped feature. Through Hadrus' AI development service we provide each of these along with all of our other full services including design, ops, growth, and build. Additionally, since interfaces are important to users, it is usually through quality UI/UX design that an AI feature becomes usable.

The cost to develop an artificial intelligence application can vary greatly depending on three major factors, your level of data readiness, the point at which you decide to start building your own AI applications ("the Build Ladder"), and the amount of integration and testing required for the completed product. Do not accept a quote from someone regarding the cost of developing an AI application without them first scoping the project.

The most important factors that drive costs are

• Data work: cleaned, labeled, and made accessible as a major part of the hidden cost.
• Approach: API-based build is much cheaper than custom-made or fine-tuned model.
• Integration: connecting with your current systems requires significant engineering effort.
• Evaluation: Evaluation is critical for successful implementation of any project or program; therefore, evaluating properly is an essential part of the budgeting process.
• Running costs: Costs associated with a machine learning application do not stop when it launches. These include costs related to inference (making predictions on new data), hosting (the physical hardware that hosts your application), monitoring (tracking how well the application performs), and retraining (updating the model with new data as it becomes available).

Reliability can be achieved by starting at the bottom level of potential success, proving the value of your product/service with a limited test/pilot program and then advancing up the ladder based upon the success of the test/pilot program. To obtain an estimate for your specific project/assignment, please reach out to our team.

An API may be used if the activity being performed is common; hire a partner when you have no AI engineering expertise; or develop within your organization (in-house) if AI is critical to your product and you can afford to maintain a team of engineers. Your decision should depend upon your capabilities and strategic requirements.

An API is appropriate for projects where quick turnaround time and limited involvement are desired. A partner is better suited for organizations who desire to deliver a product at a high level of quality, but do not have the resources to hire dedicated staff; this would include agency partners who perform white-label services. When AI will serve as your primary differentiator and you are able to retain top talent for the foreseeable future, in-house development is typically the best option. Many companies use a hybrid model, utilizing a partner to create the initial version of their product while simultaneously developing internal capabilities.

The majority of errors committed during a project include; failing to start with a quantifiable objective, underestimating the quality of your data, omitting evaluations and considering the commencement of a product as the culmination of the process. Each can be avoided.

Teams often over-engineer when they develop a custom model using an API when one is available; however, teams often do not adequately prepare for potential problems associated with privacy, erroneous answers (hallucinations), and bias. Teams should establish success criteria prior to beginning development work, create an evaluation data set as early as possible, use humans to review outputs when errors are particularly costly, and plan for continuous improvement. After a team has implemented an AI feature into production, the team will need to determine how to promote that feature to its target audience. This is where growth and marketing come into play.

Start by choosing one narrow, high-value problem, auditing your data, and running a four-week pilot on the lowest suitable rung of the Build Ladder. Small, measurable pilots beat large speculative programs.

A practical first month looks like this:

• Week 1: pick the problem, define the success metric, and review available data.
• Week 2: prototype with an API or retrieval approach.
• Week 3: build an evaluation set and test the prototype against it.
• Week 4: review results, estimate production cost, and decide whether to scale, adjust, or stop.