Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lancache is not caching most files #52

Open
mushi1996 opened this issue Sep 4, 2019 · 10 comments
Open

lancache is not caching most files #52

mushi1996 opened this issue Sep 4, 2019 · 10 comments
Labels
question Further information is requested

Comments

@mushi1996
Copy link

mushi1996 commented Sep 4, 2019

EDIT: So after so looking through the logs it does technically appear to cache "most" files but enough are left to to essentially remove 99% of any speed benefit.

I am having 3 problems the first one is the main one the other two are just questions:

Main problem:
lancache is not caching most steam files. I decided to try downloading a small game to test. The first time as expected the download ran a little slower (3MB/s instead of my usual 5MB/s - 8MB/s). The second time I went to download the speed went up to 5MB/s 20% "MISS". The third time it went to 15MB/s 10% "MISS", the fourth time it hit about 25MB/s 3% "MISS" and then finally on the fifth, sixth and seventh tries it capped out at 34.6MB/s <1% "MISS" which is great. The problem is if I have to download a game 5 times to get it to cache properly I haven't really saved any time over just leaving it in my original configuration.

I always redownloaded the same game without DLC or mods so that wasnt interfering with anything.

Questions:
So there was something mentioned about adding additional steam IPs? Are these actual steam server IPs or additional local IPs I can assign the server? If these are actual steam IPs could you explain where I can find these additional ones and how/where I put them. I'm sorry if it is a silly question but I can't for the life of me figure it out.

Here is a zip of all the log files cleared each run and attempted again with their DL rate in the title. I did this after I typed up the stuff above so DL rate may be slightly different.

https://www.mediafire.com/file/9nk7qz91j1iew91/steamcachelogs.zip/file

Also does this have any chance of killing my network connection? Literally to the point where unless I reapply my netplan I can no longer connect to the internet on my server because I had that happen once and I dont know if it was related or not.

Run script:
docker run --restart unless-stopped --name lancache-dns --detach -p 192.168.1.14:53:53/udp -e USE_GENERIC_CACHE=true -e LANCACHE_IP=192.168.1.14 lancachenet/lancache-dns:latest
docker run --restart unless-stopped --name lancache --detach -v /root/steam-cache:/data/cache -v /root/steam-cache:/data/logs -p 192.168.1.14:80:80 lancachenet/monolithic:latest
docker run --restart unless-stopped --name sniproxy --detach -p 192.168.1.14:443:443 lancachenet/sniproxy:latest
echo Please configure your router/dhcp server to serve dns as 192.168.1.14

*I removed the IP parsing bit and just directly put in my IP because it was not getting it properly. Also had to put my IP in for :53:53 because it was having trouble binding to the port.

Thanks.

@Candunc
Copy link

Candunc commented Sep 13, 2019

Do you have IPv6 access on your network? If so, windows may automatically use your ISP's IPv6 DNS servers and bypass Monolithic.

If you want to confirm (and I'm assuming you are on Windows) open up resource monitor and click network and expand the Network Activity tab. When downloading a game you should see Steam.exe downloading from 192.168.1.14. If it is downloading from other addresses then your DNS is incorrectly set up, or you need to flush your cache.

Capture

@unspec unspec added the help wanted Extra attention is needed label Feb 22, 2020
@unspec
Copy link
Member

unspec commented Feb 23, 2020

@mushi1996 Are you still having this problem? I cannot access that logs link.

A recent change in the steam client has added better cache support - see #85 . We would recomend trying again with the latest version and see if you still having any problems and if so reupload the logs.

@unspec unspec added the question Further information is requested label Feb 23, 2020
@VibroAxe VibroAxe added inactive Issue appears to be inactive and removed help wanted Extra attention is needed labels Jan 15, 2021
@stale stale bot removed the inactive Issue appears to be inactive label Jan 15, 2021
@bsavag3
Copy link

bsavag3 commented Feb 6, 2021

I too am experiencing this same issue. I've tried downloading but it does not appear to be catching the files. I am only using IPv4 on my network.

@GnB-Machine
Copy link

GnB-Machine commented Sep 26, 2022

--- Same issue: access miss content ---

Hello LAN peaply,
This month my friend and me are installed our first LAN-Cache on a virtual Ubuntu Desktop 22.04 running on a ESXi 6.7 with Xeon E5-1603 (4 core 2,6 GHz).
We builded a volume group with 3 SSDs (2 TB) and 2 HDDs (3 TB) space (12 TB totaly) and mounted it in /media/lancache
Then we installed lancache as described on your installation tutorial and installed docker, docker compose and we updated the os with it´s newest updates.
So far so good

We configured the .env as follow: (comments deleted for this post)

GNU nano 6.2 .env
USE_GENERIC_CACHE=true
LANCACHE_IP=192.168.206.206 192.168.206.207 192.168.206.208
DNS_BIND_IP=192.168.206.206
UPSTREAM_DNS=192.168.207.207
CACHE_ROOT=/media/lancache
CACHE_DISK_SIZE=11700000m
CACHE_INDEX_SIZE=3500m
CACHE_MAX_AGE=3650d
TZ=Europe/London

The LAN-Cache-DNS is the primary and only DNS-Server in our local network. It works fine resolveing DNS.

gnbadmin@lan-cache:~/lancache$ sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
08efbbd6038d lancachenet/monolithic:latest "/bin/bash -e /init/…" 7 days ago Up 2 hours 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp lancache_monolithic_1
03f0ed04be9b lancachenet/lancache-dns:latest "/bin/bash -e /init/…" 7 days ago Up 2 hours 192.168.206.206:53->53/tcp, 192.168.206.206:53->53/udp lancache_dns_1

We startet both containers and began to test downloading STEAM games.
After some tests we observed that not every game was beeing cached.
As I observed big games greater than 8 or 10 Gigabytes space where not cached.
Sometimes it "HIT", most times no.

[steam] 192.168.207.5 / - - - [26/Sep/2022:16:01:01 +0100] "GET /depot/731/chunk/7aec7769a64dda6fde17cb3f4c6ac422b478bbfe HTTP/1.1" 200 120000 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache5-fra2.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:01:01 +0100] "GET /depot/731/chunk/2ad6c3a77300d2d24feabdcbe409fa06defaeaf3 HTTP/1.1" 200 179392 "-" "Valve/Steam HTTP Client 1.0" "HIT" "cache2-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:01:01 +0100] "GET /depot/731/chunk/5c0e7aebad980a0c15555d6568698d7694cc6a3f HTTP/1.1" 200 1037680 "-" "Valve/Steam HTTP Client 1.0" "HIT" "cache2-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:01:01 +0100] "GET /depot/731/chunk/e0349735455b964bac4b092f04b29ea3376e420c HTTP/1.1" 200 112 "-" "Valve/Steam HTTP Client 1.0" "HIT" "cache2-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:01:01 +0100] "GET /depot/731/chunk/bb7d6eb8ced88833fb2dc93d8e6615a5002f9e48 HTTP/1.1" 200 278048 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache3-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:01:01 +0100] "GET /depot/731/chunk/5f752088196cc00206601321d542824a4db21fa9 HTTP/1.1" 200 1044240 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:01:01 +0100] "GET /depot/731/chunk/4a99208fb5ff54c1337ec81d7a349bd4f553191b HTTP/1.1" 200 316960 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache3-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:01:01 +0100] "GET /depot/731/chunk/7f01b3206a0b5350dff8d9dc891933c94b3b7481 HTTP/1.1" 200 663760 "-" "Valve/Steam HTTP Client 1.0" "HIT" "cache2-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:01:01 +0100] "GET /depot/731/chunk/8bfb7eca8452676f700b782bdf5eb70607f5ab7e HTTP/1.1" 200 267568 "-" "Valve/Steam HTTP Client 1.0" "HIT" "cache3-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:01:01 +0100] "GET /depot/731/chunk/87e7672da468a26505417ca7f137690a12423081 HTTP/1.1" 200 226352 "-" "Valve/Steam HTTP Client 1.0" "HIT" "cache5-fra2.steamcontent.com" "-"
gnbadmin@lan-cache:/media/lancache/logs$ tail access.log
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/ea4d93f9bd8768dacd32ad539595a3232f14aa44 HTTP/1.1" 200 847040 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache3-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/c3337fe52776f8599cf782b37764650bf9fd9f7c HTTP/1.1" 200 947840 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/e3d8166783368d9911d5260fcccd9878905f809a HTTP/1.1" 200 883120 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/ace235ceb9430725f40a902f278e9d07510ff8fc HTTP/1.1" 200 891616 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache13-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/776745feea5eb1f0779db3e52d709b0b1a2925d3 HTTP/1.1" 200 764768 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache5-fra2.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/c49dc488eccf37ceb45ea8430460ec66ebcbd3f1 HTTP/1.1" 200 970528 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/34465dd5451b13cf4f84326ef9c8f2787683aa86 HTTP/1.1" 200 923696 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache3-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/a216c82b350ff0579f23e3532fa42c775c114e31 HTTP/1.1" 200 958928 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/e57931434b19dcbf362a250221c296ab8d207118 HTTP/1.1" 200 1026080 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache13-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:25 +0100] "GET /depot/731/chunk/e3aadb0583deff41b9b7b47b015831e59de63ca8 HTTP/1.1" 200 1016944 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache5-fra2.steamcontent.com" "-"
gnbadmin@lan-cache:/media/lancache/logs$ tail access.log
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/ea4d93f9bd8768dacd32ad539595a3232f14aa44 HTTP/1.1" 200 847040 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache3-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/c3337fe52776f8599cf782b37764650bf9fd9f7c HTTP/1.1" 200 947840 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/e3d8166783368d9911d5260fcccd9878905f809a HTTP/1.1" 200 883120 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/ace235ceb9430725f40a902f278e9d07510ff8fc HTTP/1.1" 200 891616 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache13-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/776745feea5eb1f0779db3e52d709b0b1a2925d3 HTTP/1.1" 200 764768 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache5-fra2.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/c49dc488eccf37ceb45ea8430460ec66ebcbd3f1 HTTP/1.1" 200 970528 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/34465dd5451b13cf4f84326ef9c8f2787683aa86 HTTP/1.1" 200 923696 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache3-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/a216c82b350ff0579f23e3532fa42c775c114e31 HTTP/1.1" 200 958928 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/e57931434b19dcbf362a250221c296ab8d207118 HTTP/1.1" 200 1026080 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache13-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:25 +0100] "GET /depot/731/chunk/e3aadb0583deff41b9b7b47b015831e59de63ca8 HTTP/1.1" 200 1016944 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache5-fra2.steamcontent.com" "-"
gnbadmin@lan-cache:/media/lancache/logs$ tail access.log
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/ea4d93f9bd8768dacd32ad539595a3232f14aa44 HTTP/1.1" 200 847040 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache3-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/c3337fe52776f8599cf782b37764650bf9fd9f7c HTTP/1.1" 200 947840 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/e3d8166783368d9911d5260fcccd9878905f809a HTTP/1.1" 200 883120 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/ace235ceb9430725f40a902f278e9d07510ff8fc HTTP/1.1" 200 891616 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache13-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/776745feea5eb1f0779db3e52d709b0b1a2925d3 HTTP/1.1" 200 764768 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache5-fra2.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/c49dc488eccf37ceb45ea8430460ec66ebcbd3f1 HTTP/1.1" 200 970528 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-ams1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/34465dd5451b13cf4f84326ef9c8f2787683aa86 HTTP/1.1" 200 923696 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache3-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/a216c82b350ff0579f23e3532fa42c775c114e31 HTTP/1.1" 200 958928 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache2-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:24 +0100] "GET /depot/731/chunk/e57931434b19dcbf362a250221c296ab8d207118 HTTP/1.1" 200 1026080 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache13-fra1.steamcontent.com" "-"
[steam] 192.168.207.5 / - - - [26/Sep/2022:16:02:25 +0100] "GET /depot/731/chunk/e3aadb0583deff41b9b7b47b015831e59de63ca8 HTTP/1.1" 200 1016944 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cache5-fra2.steamcontent.com" "-"

I´m want to understand the reason / background of this behavior.
I googeled this issue and fond other guys with the same problem but no solution.
May be we made something wrong but I don´t know what?

If you have a tip I would be happy. Please tell me if you need further details.
Thanks and Greetings Machine -> visit us on our LAN-Party "www.gunsnbits.de"
; )

@VibroAxe
Copy link
Member

VibroAxe commented Sep 26, 2022

@GnB-Machine what dns settings have you pushed to the clients? They should be only assigned 192.168.206.206 not any backup/secondary?

What are you doing in your test, i would expect the following workflow

  • Download game on client a (initial preload, miss in log) - Should see speeds approaching internet speed
  • Delete game on client a
  • Download game on client a (cached data, hit in log) - Should see speeds increase up to io limits
  • Download game on client b (cached data, hit in log) - Should see speeds up to io limits

The miss in the logs means the cache does not currently have that section of the file and needs to grab it from upstream, it will only serve cached traffic on the 2nd attempt

@GnB-Machine
Copy link

Hello Vibro,

thanks a lot for your quick response.
OK, sounds logical.
I thought it relates to the downloaded content direct from steam.
So I have to test it further and ceck the internet bandwidth again aftern installing secound time from another client.

Regarding DNS-Server
We do not propagate a second DNS-Server via DHCP. 192.168.206.206 is the primary one.
But it forwards the requests to its forward-DNS (192.168.207.207) how is locatet in the same network.
I know its a little bit unusual but we additionally using a pi-hole.
I have whitelisted every STEAM, Origin, Battlenet .... URL you are providing on github to be shure do not running in a black hole.

Thanks and Greetings Machine

@GnB-Machine
Copy link

You have to visit us this weekend
www.gunsbits.de
belive me
You will never forget
; )

@IFD3
Copy link

IFD3 commented Oct 20, 2022

I have similar issues. Testing small games (downloading, deleting, redownloading) works, but the longer I test the more MISSes appear until nothing gets a HIT. Well what I found out is that after restarting the docker container it seems that the cached is indexed again and everything is a HIT.
At least thats how it feels like.
But restarting the container often or before expecting HITs is not really what I wanted.

@Elkropac
Copy link

Elkropac commented Oct 30, 2022

Hi, i'm observing high MISS rate too.
Yesterday, my son installed Chilvary2 from game pass, i got around 5000 lines on file
/11/a242a63a-bf15-4bfc-8baa-e15ccf859e46/14a4a2bb-8454-4753-bc7a-3327ee1b343f/1.0.27.0.f848152d-2d44-4e89-b1ab-2dabd9c9d8c4/TripwireInteractiveLLC.Chivalry2_1.0.27.0_x64__237dqjpew1r3m.msixvc
in lancache access.log, most of them are MISS (as expected).

Today, i tried to install it too.
I got around 20000 lines with MISS and also around 5000 HITs.
Why so many misses? I was observing download speeds around 4-5MB/s, which is my WAN connections speed.
It sometimes went to 20-60MB/s, when it was HITting lancache, but most of the time, i was on my WAN speed.

My network has ipv6, but all my dns servers are ipv4

@mikecummings34

This comment was marked as off-topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

9 participants