Offline enviroment, how to run with local Ollama models?

[TRYOKETHEPEN]

Describe the bug

Run with docker raise error: read ECONNRESET
The error relates to network problem while access open router model, but i didnt set its API
I only want to use local ollama, what should i do

Link to the Bolt URL that caused the error

\

Steps to reproduce

1. My server is offline, only can access our internal registry and github
2. My server's feature: Ubuntu20.04, x64
3. cd /home/containers
4. git clone https://github.com/stackblitz-labs/bolt.diy
5. cd bolt.diy
6. nano Dockerfile:
7. Modify the first few lines as: (xxxxx is for secrecy)

# change to our internal registry
ARG BASE=xxxxx/node:20.18.0
FROM ${BASE} AS base

# make sure npm registry success
USER root
WORKDIR /app

# Install dependencies (this step is cached as long as the dependencies don't change)
COPY package.json pnpm-lock.yaml ./

# change to our internal registry
RUN npm config set registry http://xxxxx/npm-official/
# corepack enable pnpm does not obey the registry, so change it to npm install
RUN npm install -g pnpm
RUN pnpm install
# RUN corepack enable pnpm && pnpm install

8. npm run dockerbuild
9. docker-compose --profile development up
10. Access localhost:5173, raise Error:

(base) root@AIServer:/home/containers/bolt.diy# docker-compose --profile development up
WARN[0000] The "GROQ_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "HuggingFace_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "OPENAI_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "ANTHROPIC_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "OPEN_ROUTER_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "GOOGLE_GENERATIVE_AI_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "OLLAMA_API_BASE_URL" variable is not set. Defaulting to a blank string. 
WARN[0000] The "TOGETHER_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "TOGETHER_API_BASE_URL" variable is not set. Defaulting to a blank string. 
WARN[0000] The "GROQ_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "HuggingFace_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "OPENAI_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "ANTHROPIC_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "OPEN_ROUTER_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "GOOGLE_GENERATIVE_AI_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "OLLAMA_API_BASE_URL" variable is not set. Defaulting to a blank string. 
WARN[0000] The "TOGETHER_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "TOGETHER_API_BASE_URL" variable is not set. Defaulting to a blank string. 
WARN[0000] Found orphan containers ([bolt-ai]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
[+] Running 1/0
 ? Container boltdiy-app-dev-1  Created                                                                                                                                                         0.0s 
Attaching to app-dev-1
app-dev-1  | 
app-dev-1  | > [email protected] dev /app
app-dev-1  | > node pre-start.cjs  && remix vite:dev "--host" "0.0.0.0"
app-dev-1  | 
app-dev-1  | 
app-dev-1  | ★═══════════════════════════════════════★
app-dev-1  |           B O L T . D I Y
app-dev-1  |          ??  Welcome  ??
app-dev-1  | ★═══════════════════════════════════════★
app-dev-1  | 
app-dev-1  | ?? Current Commit Version: 50e677878446f622531123b19912f38e8246afbd
app-dev-1  | ★═══════════════════════════════════════★
app-dev-1  | [warn] Data fetching is changing to a single fetch in React Router v7
app-dev-1  | ┃ You can use the `v3_singleFetch` future flag to opt-in early.
app-dev-1  | ┃ -> https://remix.run/docs/en/2.13.1/start/future-flags#v3_singleFetch
app-dev-1  | ┗
app-dev-1  |   ?  Local:   http://localhost:5173/
app-dev-1  |   ?  Network: http://10.1.8.2:5173/
app-dev-1  | TypeError: fetch failed
app-dev-1  |     at node:internal/deps/undici/undici:13185:13
app-dev-1  |     at processTicksAndRejections (node:internal/process/task_queues:95:5)
app-dev-1  |     at Object.getOpenRouterModels [as getDynamicModels] (/app/app/utils/constants.ts:574:5)
app-dev-1  |     at async Promise.all (index 2)
app-dev-1  |     at Module.initializeModelList (/app/app/utils/constants.ts:654:7)
app-dev-1  |     at handleRequest (/app/app/entry.server.tsx:30:3)
app-dev-1  |     at handleDocumentRequest (/app/node_modules/.pnpm/@[email protected][email protected]/node_modules/@remix-run/server-runtime/dist/server.js:340:12)
app-dev-1  |     at requestHandler (/app/node_modules/.pnpm/@[email protected][email protected]/node_modules/@remix-run/server-runtime/dist/server.js:160:18)
app-dev-1  |     at /app/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]_typ_3djlhh3t6jbfog2cydlrvgreoy/node_modules/@remix-run/dev/dist/vite/cloudflare-proxy-plugin.js:70:25 {
app-dev-1  |   [cause]: Error: read ECONNRESET
app-dev-1  |       at TLSWrap.onStreamRead (node:internal/stream_base_commons:218:20)
app-dev-1  |       at TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
app-dev-1  |     errno: -104,
app-dev-1  |     code: 'ECONNRESET',
app-dev-1  |     syscall: 'read'
app-dev-1  |   }
app-dev-1  | }
app-dev-1  | TypeError: fetch failed
app-dev-1  |     at node:internal/deps/undici/undici:13185:13
app-dev-1  |     at processTicksAndRejections (node:internal/process/task_queues:95:5)
app-dev-1  |     at Object.getOpenRouterModels [as getDynamicModels] (/app/app/utils/constants.ts:574:5)
app-dev-1  |     at async Promise.all (index 2)
app-dev-1  |     at Module.initializeModelList (/app/app/utils/constants.ts:654:7)
app-dev-1  |     at handleRequest (/app/app/entry.server.tsx:30:3)
app-dev-1  |     at handleDocumentRequest (/app/node_modules/.pnpm/@[email protected][email protected]/node_modules/@remix-run/server-runtime/dist/server.js:390:14)
app-dev-1  |     at requestHandler (/app/node_modules/.pnpm/@[email protected][email protected]/node_modules/@remix-run/server-runtime/dist/server.js:160:18)
app-dev-1  |     at /app/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected][email protected]_typ_3djlhh3t6jbfog2cydlrvgreoy/node_modules/@remix-run/dev/dist/vite/cloudflare-proxy-plugin.js:70:25 {
app-dev-1  |   [cause]: Error: read ECONNRESET
app-dev-1  |       at TLSWrap.onStreamRead (node:internal/stream_base_commons:218:20)
app-dev-1  |       at TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
app-dev-1  |     errno: -104,
app-dev-1  |     code: 'ECONNRESET',
app-dev-1  |     syscall: 'read'
app-dev-1  |   }
app-dev-1  | }

Expected behavior

\

Screen Recording / Screenshot

as above

Platform

as above

Provider Used

Ollama

Model Used

\

Additional context

No response

[TRYOKETHEPEN]

[1] The error above is trying to access https://openrouter.ai/api/v1/models
[2] I download the models.json file at online enviroment
[3] Then I changed the constants.ts as:

async function getOpenRouterModels(): Promise<ModelInfo[]> {
  // const data: OpenRouterModelsResponse = await (
  //   await fetch('https://openrouter.ai/api/v1/models', {
  //     headers: {
  //       'Content-Type': 'application/json',
  //     },
  //   })
  // ).json();

  const data: OpenRouterModelsResponse = JSON.parse(fs.readFileSync('/home/containers/bolt.diy/models.json', 'utf8'));

  return data.data
    .sort((a, b) => a.name.localeCompare(b.name))
    .map((m) => ({
      name: m.id,
      label: `${m.name} - in:$${(m.pricing.prompt * 1_000_000).toFixed(
        2,
      )} out:$${(m.pricing.completion * 1_000_000).toFixed(2)} - context ${Math.floor(m.context_length / 1000)}k`,
      provider: 'OpenRouter',
      maxTokenAllowed: 8000,
    }));
}

[4] Now I can open the localhost:5173 web page after npm run dev (Docker should be considered after local running ok)
[5] But I wonder how to load my local ollama model (http://localhost:11434)
[6] I've checked my local ollama model is correct via http://localhost:11434/v1/models

and via http://localhost:11434/api/tags

[7] I've tried add "http://localhost:11434" to Settings-Providers-Ollama-Base URL, or to .env.local file, but it didnt work.
The model label select box is empty, and chat raise error:

[TRYOKETHEPEN]

[1] I changed the constants.ts PROVIDER_LIST as:

{
    name: 'Ollama',
    staticModels: [{
      name: 'qwen2.5-coder-14b-instruct-q4_0:latest',
      label: 'qwen2.5-coder-14b-instruct-q4_0:latest',
      provider: 'Ollama',
      maxTokenAllowed: 8000,
    }],
    // getDynamicModels: getOllamaModels,
    // getApiKeyLink: 'https://ollama.com/download',
    // labelForGetApiKey: 'Download Ollama',
    // icon: 'i-ph:cloud-arrow-down',
  },

[2] Then I can select my local model, but chat raise error:

[thecodacus]

its fine these error wont cause the bolt to stop working. these are just alerts for you to know that its unable to load dynamic models.

the only concern about fully offline is that you wont be able to load the webcontainer as this requires us to pull some web assembly from stackBlitz server, and not available offline. once its loaded you can actually work offline but the initial load requires access to StackBlitz assets

[TRYOKETHEPEN]

its fine these error wont cause the bolt to stop working. these are just alerts for you to know that its unable to load dynamic models.

the only concern about fully offline is that you wont be able to load the webcontainer as this requires us to pull some web assembly from stackBlitz server, and not available offline. once its loaded you can actually work offline but the initial load requires access to StackBlitz assets

Thanks for your reply!

[1] I tried Chrome canary, find out that there is no need to add staticModels in PROVIDER_LISTS at constans.ts. It behaves as: can select ollama model but /api/chat raise error.

(Note that the model label is auto generated, endwith parameter size.)
[2] I tried the newest bolt.diy, still got the same problem.
[3] I traced the error, access /api/chat is finally excuted in UseChat at @ai-sdk/react , but I find out that there is no @ai-sdk/react pack at node_modules, it cannot be installed by pnpm install (strange).
[4] Then I manually npm install @ai-sdk/[email protected] , but the /api/chat error still exist.

By chance, I tried npm run build, npm run start, chat no error any more.

I dont know whats the difference between dev and build .
Now my problem is as what you write ---- can chat but cannot use webcontainer feature:

[thecodacus]

yes the webcontainer wont start without loading the web assembly from stackblitz.
and as webcontainer is a proprietary software from stackblitz we cannot have a the assembly files offline and dont have access to those files

[Jungly78]

So i manged to make this work out now its not using my 4090 it uses cpu but when i use openweui it uses 100% gpu i tried to modify .env but doesent help any one have a clue

iam using windows as operating system and installed ollama on my desktop and using docker for bolt, accesing bolt from destop to laptop remotly, tried installing bolt both on desktop and laptop having same issue fro some reason it wont use gpu and when i install on laptop i uses cpu there even not needed, please help am i missing something here , for information iam tottaly noob on this so please have patience

[thecodacus]

need to understand your full setup you are using

[Jungly78]

1. Operating System:
   Desktop: Windows 11 with NVIDIA RTX 4090 (Ollama installed here).
   Laptop: Windows 10 (accessing Bolt remotely).
   Machines are on diffrent network.
2. Software Installation:
   Ollama: Installed on the desktop (Windows) using the standard installer.
   Bolt (Docker):
   Initially installed and running on the desktop via Docker.
   Also tested installing Bolt on the laptop via Docker, but had the same issue.
3. Issue Details:
   When I run Open WebUI (also docker installation) on the desktop, it uses 100% GPU (as expected).
   When I use Bolt (Docker):
   It always uses CPU, even though Ollama (on the same desktop) is configured to use the GPU.
   On the laptop, when accessing Bolt remotely, it still defaults to the CPU.
   I modified the .env file for Bolt with the following:
   env
   Kopiera kod
   OLLAMA_API_BASE_URL=http://:11434
   OLLAMA_USE_GPU=1
   DEFAULT_NUM_CTX=2048
   But it doesn't change the behavior.
4. Observations:
   GPU usage works fine with Open WebUI, so I assume Ollama's GPU setup is correct.
   Bolt does not seem to pass the GPU flag or some configuration correctly to Ollama.
5. Things I Tried:
   Tested Bolt on both desktop and laptop with the same results (CPU-only).
   Verified that Ollama's API is accessible from both machines (laptop and desktop).
   Monitored GPU usage with nvidia-smi—Bolt does not utilize the GPU at all.
6. My Questions:
   Is there a specific configuration or flag I need to set in Bolt to ensure GPU usage?
   Could Docker or a network configuration be interfering with Bolt's ability to utilize the GPU on the desktop?
   Any debugging tips or steps to ensure Bolt passes the correct GPU settings to Ollama?
   I hope this provides enough context. Please let me know if there’s anything else I can clarify. Thank you for your help and patience!

[Coding-xiaobo]

@TRYOKETHEPEN Bro, I use bolt.new in a similar environment as you. After interacting with llm on the left, can the file system on the right generate files?

[Jungly78]

Yes but that was not the problem problem was it was not utilizing gpu 100%

when i used commande ollama ps

it gave me

NAME ID SIZE PROCESSOR UNTIL
qwen2.5-coder:32b 4bd6cbf2d094 32 GB 28%/72% CPU/GPU Forever

so i had to modify .env file

Loaded environment variables:
OLLAMA_API_BASE_URL: http://my.adress.com:11434
OLLAMA_USE_GPU: 0
DEFAULT_NUM_CTX: 6144

install
pnpm install dotenv

and change

vite.config.ts like so

import { cloudflareDevProxyVitePlugin as remixCloudflareDevProxy, vitePlugin as remixVitePlugin } from '@remix-run/dev';
import UnoCSS from 'unocss/vite';
import { defineConfig, type ViteDevServer } from 'vite';
import { nodePolyfills } from 'vite-plugin-node-polyfills';
import { optimizeCssModules } from 'vite-plugin-optimize-css-modules';
import tsconfigPaths from 'vite-tsconfig-paths';
import dotenv from 'dotenv'; // Import dotenv för att ladda miljövariabler

// Ladda .env-variabler
dotenv.config();

console.log('Loaded environment variables:');
console.log('OLLAMA_API_BASE_URL:', process.env.OLLAMA_API_BASE_URL || 'Not set');
console.log('OLLAMA_USE_GPU:', process.env.OLLAMA_USE_GPU || 'Not set');
console.log('DEFAULT_NUM_CTX:', process.env.DEFAULT_NUM_CTX || 'Not set');

export default defineConfig((config) => {
return {
build: {
target: 'esnext',
},
plugins: [
nodePolyfills({
include: ['path', 'buffer'],
}),
config.mode !== 'test' && remixCloudflareDevProxy(),
remixVitePlugin({
future: {
v3_fetcherPersist: true,
v3_relativeSplatPath: true,
v3_throwAbortReason: true,
v3_lazyRouteDiscovery: true,
},
}),
UnoCSS(),
tsconfigPaths(),
chrome129IssuePlugin(),
config.mode === 'production' && optimizeCssModules({ apply: 'build' }),
],
envPrefix: ["VITE_", "OPENAI_LIKE_API_", "OLLAMA_API_BASE_URL", "LMSTUDIO_API_BASE_URL", "TOGETHER_API_BASE_URL"],
css: {
preprocessorOptions: {
scss: {
api: 'modern-compiler',
},
},
},
};
});

function chrome129IssuePlugin() {
return {
name: 'chrome129IssuePlugin',
configureServer(server: ViteDevServer) {
server.middlewares.use((req, res, next) => {
const raw = req.headers['user-agent']?.match(/Chrom(e|ium)/([0-9]+)./);

    if (raw) {
      const version = parseInt(raw[2], 10);

      if (version === 129) {
        res.setHeader('content-type', 'text/html');
        res.end(
          '<body><h1>Please use Chrome Canary for testing.</h1><p>Chrome 129 has an issue with JavaScript modules & Vite local development, see <a href="https://github.com/stackblitz/bolt.new/issues/86#issuecomment-2395519258">for more information.</a></p><p><b>Note:</b> This only impacts <u>local development</u>. `pnpm run build` and `pnpm run start` will work fine in this browser.</p></body>',
        );

        return;
      }
    }

    next();
  });
},

};
}

And now i get

C:\Users\12334>ollama ps
NAME ID SIZE PROCESSOR UNTIL
qwen2.5-coder:32b 4bd6cbf2d094 22 GB 100% GPU Forever

and its fast :)))

[TRYOKETHEPEN]

@TRYOKETHEPEN Bro, I use bolt.new in a similar environment as you. After interacting with llm on the left, can the file system on the right generate files?

你好，它会一直卡在生成文件步骤上。
如 @thecodacus 上面所述，生成文件、执行命令等webcontainer特性 需要连接stackblitz服务器下载一些组件才能实现。我对web不太懂，暂时放弃这个了，后续如果有进展，希望你可以分享一下

[Coding-xiaobo]

@TRYOKETHEPEN Bro, I use bolt.new in a similar environment as you. After interacting with llm on the left, can the file system on the right generate files?

你好，它会一直卡在生成文件步骤上。 如 @thecodacus 上面所述，生成文件、执行命令等webcontainer特性 需要连接stackblitz服务器下载一些组件才能实现。我对web不太懂，暂时放弃这个了，后续如果有进展，希望你可以分享一下

后来我发现是代理问题，代理导致我拉取webcontainer相关依赖失败了，所以那个文件系统起不来，处理好代理之后，我这边文件系统正常了，但是和终端交互不正常