Lesson · 40 min · Free
Running LLMs Locally and via APIs
Running LLMs Locally and via APIs body { font-family: sans-serif; line-height: 1.6; margin: 20px; } h1, h2 { color: #2c3e50; } pre { background-color: #ecf0f1; padding: 15px; border-radius: 5px; overflow-x: auto; } code
Running LLMs Locally and via APIs
Welcome to this crucial lesson in your LLM Engineering Bootcamp! As pharmacy and biotech professionals, understanding how to interact with Large Language Models (LLMs) is paramount. This lesson will demystify the two primary methods of deploying and utilizing LLMs: running them locally on your own hardware and accessing them remotely via Application Programming Interfaces (APIs). Each approach has distinct advantages and disadvantages, particularly concerning data privacy, computational resources, and development flexibility, all of which are critical considerations in regulated environments. The choice between local execution and API access often boils down to a trade-off between control and convenience. Local execution offers unparalleled control over data, security, and model customization, making it ideal for sensitive patient data or proprietary research. However, it demands significant computational resources and technical expertise. Conversely, API-based access provides convenience, scalability, and access to state-of-the-art models without the overhead of infrastructure management, but it involves entrusting your data to a third-party provider and incurring usage costs.
Local LLM Deployment: Control and Privacy
Running an LLM locally means downloading the model weights and executing the inference process directly on your computer or a dedicated server. This approach is highly valued in fields like pharmacy and biotech where data privacy and intellectual property are paramount. By keeping data processing in-house, you minimize the risk of data breaches and comply more easily with regulations like HIPAA or GDPR. However, local deployment requires substantial hardware. Modern LLMs, even smaller variants, can demand gigabytes of RAM and powerful GPUs (Graphics Processing Units) for reasonable inference speeds. For example, a 7B parameter model might require 8-16GB of VRAM, while larger models can demand 24GB or more. Several frameworks facilitate local LLM deployment. Hugging Face's Transformers library is a leading choice, providing a unified interface for hundreds of pre-trained models. Libraries like llama.cpp (for CPU-based inference) or frameworks leveraging NVIDIA's CUDA for GPU acceleration are also popular for optimizing local performance. The process typically involves installing the necessary libraries, downloading the desired model (often in a quantized format for efficiency), and writing a script to load the model and generate responses. Here's a simplified Python example demonstrating local inference using the Hugging Face Transformers library. Note that this assumes you have a model downloaded or accessible from the Hugging Face Hub, and sufficient hardware. from transformers import pipeline # Load a pre-trained model for text generation locally # You might need to install 'torch' or 'tensorflow' and 'transformers' # e.g., pip install transformers torch sentencepiece # For a small, instruction-tuned model, e.g., 'distilbert-base-uncased' or 'gpt2' # For larger models like 'mistralai/Mistral-7B-Instruct-v0.2', # ensure you have enough RAM/VRAM and potentially install 'accelerate' and 'bitsandbytes' # For demonstration, we'll use a smaller, readily available model. generator = pipeline('text-generation', model='distilgpt2') prompt = "Pharmacological interventions for type 2 diabetes include" print(f"Prompt: {prompt}\n") # Generate text # max_new_tokens controls the length of the generated output # num_return_sequences allows generating multiple different completions # Temperature can be adjusted for creativity (higher) or determinism (lower) outputs = generator(prompt, max_new_tokens=50, num_return_sequences=1, temperature=0.7) for i, output in enumerate(outputs): print(f"Generated text {i+1}:\n{output['generated_text']}\n") # Example for a more advanced, larger model (conceptual, requires significant resources) # from transformers import AutoModelForCausalLM, AutoTokenizer # import torch # # model_name = "mistralai/Mistral-7B-Instruct-v0.2" # tokenizer = AutoTokenizer.from_pretrained(model_name) # model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto") # # messages = [ # {"role": "user", "content": "What are the common side effects of metformin?"} # ] # # encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt") # # model_inputs = encodeds.to("cuda") # or "cpu" if no GPU # # generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=True) # decoded = tokenizer.batch_decode(generated_ids) # print(decoded[0])
API-Based LLM Access: Convenience and Scalability
Accessing LLMs via APIs (Application Programming Interfaces) is the most common method for many applications due to its convenience and scalability. Major providers like OpenAI (GPT series), Google (Gemini, PaLM), Anthropic (Claude), and others offer powerful LLMs as a service. You send your input (prompt) to their servers, and they return the model's output. This eliminates the need for managing hardware, software dependencies, and model updates, allowing you to focus purely on application development. For pharmacy and biotech, API access is excellent for tasks that don't involve highly sensitive data, such as general literature review, drafting non-confidential reports, or educational tools. The primary considerations here are cost (usage is typically billed per token), data privacy policies of the API provider, and potential latency issues. Always review the terms of service, especially regarding data retention and how your inputs might be used for model training. Here's a Python example using OpenAI's API. This requires an API key, which you would obtain from their platform. import openai # Replace with your actual API key # It's best practice to load this from environment variables, not hardcode it. # e.g., import os; openai.api_key = os.getenv("OPENAI_API_KEY") openai.api_key = "YOUR_OPENAI_API_KEY" def get_completion_from_messages(messages, model="gpt-3.5-turbo", temperature=0.7): """ Sends a list of messages to the OpenAI API and returns the completion. """ try: response = openai.ChatCompletion.create( model=model, messages=messages, temperature=temperature, # Controls randomness: 0.0 (deterministic) to 1.0 (very creative) ) return response.choices[0].message["content"] except Exception as e: print(f"An error occurred: {e}") return None # Example usage for a pharmacy-related query messages = [ {"role": "system", "content": "You are a helpful assistant for pharmaceutical research."}, {"role": "user", "content": "Summarize the mechanism of action of sartans in hypertension management."} ] print(f"Prompt: {messages[1]['content']}\n") response_content = get_completion_from_messages(messages) if response_content: print(f"API Response:\n{response_content}\n") # Another example: Drug interaction query messages_interaction = [ {"role": "system", "content": "You are a clinical pharmacist assistant."}, {"role": "user", "content": "What are the significant drug-drug interactions between Warfarin and Trimethoprim/Sulfamethoxazole?"} ] response_interaction = get_completion_from_messages(messages_interaction, model="gpt-4", temperature=0.5) if response_interaction: print(f"API Response (Drug Interaction):\n{response_interaction}\n") Important Note on API Keys: Never expose your API keys directly in publicly accessible code or commit them to version control. Use environment variables or secure configuration management systems to store and retrieve them.
Key Takeaways:
Local LLMs: Offer maximum data privacy and control, ideal for sensitive biotech/pharmacy data. Require significant computational resources (CPU, GPU, RAM) and technical setup. API-Based LLMs: Provide convenience, scalability, and access to cutting-edge models without infrastructure management. Involve third-party data processing and incur usage costs. Hardware Requirements: Local LLMs demand powerful GPUs with ample VRAM; smaller models might run on CPUs but slowly. Data Security: A primary differentiator. Local deployment keeps data in-house; API access means data transits to and is processed by the provider. Cost: Local LLMs have an upfront hardware cost; API LLMs have per-token usage costs. Flexibility: Local models can be fine-tuned or customized more easily. API models offer less customization but are easier to integrate.
Practice Exercise:
Imagine you are developing a prototype AI assistant for a hospital pharmacy. You need to decide whether to use a locally deployed LLM or an API-based LLM for two distinct tasks: (1) generating patient-specific medication counseling sheets based on their electronic health records (EHR), and (2) providing general, non-patient-specific information about new drug approvals from public FDA databases. For each task, justify your choice of deployment method, considering factors like data privacy, cost, computational resources, and development speed. Briefly outline how you would approach setting up each chosen method.
Watch the full lesson — free
This topic is part of The Complete LLM Engineering Bootcamp, a complete AI-narrated video course. Press play once and watch the entire lecture like a movie.
Start the course free →