Data Migration Documentation

[l-arnold]

focusing on the "-u all -d" commands to be sure that Databases are properly upgraded when imported.

I would expect Odoo does this by default in database manager, but need to confirm

Have some other references to this but just lost track of them and need to refind (were on first version of this post that was lost)

[DocCyblade]

Should have some good documentation regarding TKLBAM, as from one turnkey system to the same this should be pretty good as long as the databases get dumped and backed up. The issue would be say TKL Odoo v14 to v15. Since that's not yet out I would not worry about that just yet. I have seen in the past that TKL basically more often that not references upstream documentation on upgrades.

I had some first hand knowledge a few years ago when I went from one redline version to another with TKLBAM. The restore went as planned however, I had to manually update the database and other parts.

I would love to see a nice "easy" upgrade but to be honest it's not that simple to make something like that. That is why it's best when building a TKL appliance, try to follow the upstream install verbatim and configuration. This way it should be pretty easy to do the upgrade following upstream documentation since we followed all the guidelines etc. (So things should be where the documentation says etc)

That is my 2c but for the offical response I'll leave that up to @JedMeister 😄

[JedMeister]

I recall posting a fair bit of info re TKLBAM somewhere here recently!? I think that it may have been on our tracker mega-thread! 😄 Or was that in my head? IDK...! ❓

Yeah it'd be awesome to have hooks in place to support TKLBAM upgrades etc. Like you say Ken it's doable; but a little tricky. FWIW currently the only appliance that has that sort of logic currently is Redmine.

👍 on following conventions. Ideally use upstream conventions and/or turnkey conventions. Then people should be able to work out what is going on...

[DocCyblade]

FWIW: I did some quick reading, Odoo/OpenERP conversions from v7 to v8 is not very straight forward. So I would not put too much thought in making the TKL version with any auto upgrade functionality. As far at TKLBAM just need to make sure anywhere user data gets stored gets backed up, and also any databases that get created are also backed up. You should not need to backup any of the application, but should backup config files for sure as well.

I have used TKLBAM to create a "test" server from the production before performing a big upgrade. That way to if the upgrade goes really bad you should be able to get the production server backup and running pretty quickly from a TKLBAM backup.

Again just need to make sure that all files and data that the application generations and stores is included.

[l-arnold]

We will not try to Move from OpenERP 7 to Odoo 8. Very complicated. There are ways but..

I am only wanting to be sure that "Odoo 8 data" gets updated" (at least via script) if one Pulls an older Odoo 8 Databse into a current build. As we are using Git, the "git base" is a moving target. That Update Process, to my understanding should make the Database have all needed components be updated.

My update has gone fine. (about 2 months change) and I had a similar update a few months earlier.

Just wanting to be sure some reef doesn't show up. Mainly "personal issue". However, I think one Goal is to get Other Odoo Users to jump over as there are so many advantages to working within the TKL framework. IMHO but more than that.

[l-arnold]

(UPDATE to Directory from Database) Also, I believe the files have "hot git" within them so a User may go to them and issue a "Git Pull" which will update the code base pretty directly. Need to do it one directory at a time.

Ideally we would have some Webmin Scripts to handle subjects like this. Probably not needed in first release.

[l-arnold]

Also. Odoo 8 will likely be a more and more stable base. My read is that many users want to settle on Odoo 8 and not go to 9 as Odoo (company) is trying to commercialize it more.

Interesting Statistic here on Odoo.
94,105 commits
12 branches
79 releases
379 contributors

[l-arnold]

@JedMeister
(your great note was) here (though we should have this in the TKLBAM issue here really)

#11

Re TKLBAM here are the steps.

... I will post in the TKLBAM issue rather than here.
—

[l-arnold]

@DocCyblade So we need to get
openerp.init start working properly (it will "stop" now.

These are the instructions for Update after any GIT PULL Process with a working server:

...before any upgrade, always do a backup of your database. Then, do a git pull on every branch, stop your server and start it again with the option -u base in order to update the modules.

Reference: OCA/connector-magento#132

I Also have added A doc folder/file and want this in there. Right now the "stop" and "restart" parts are commented out in that it is not working in that mode. Need to understand the process for calling the proper variables.

[l-arnold]

Not seeing errors on
cd /opt/openerp (running as root)

/bin/bash openerp.init start
Shell returns "Starting openerp-server: openerp-server. "

However, like in our "firstrun inithooks" getting
*Service Unavailable

*The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

It seems there are some issues around ownership.

/var/logs/openerp-server.log (seems to be owned by openerp:root (root=group)
/run/openerp-server.pid is owned by "root:root"
/opt/openerp/openerp.init is owned by openerp:openerp (as with the rest of the files in /opt/openerp

I can run openerp.init with "su - openerp" if everything is set (on manual mod) to openerp:openerp

....
however, changing any which way still will not allow the real server to Restart without Reboot.

even tried: systemctl start openerp-server.service
(which does not return errors, but I do not get a hot Odoo Connection afterwards still)

Thoughts?

[DocCyblade]

Not sure why it works but during first boot I bounced (restarted) first stop apache2, odoo, then PostgreSQL, the. Start them on the reverse order but used systemd commands to do them. That seemed to work. I have a hunch its with the proxy part of Apache.

Have not had too much time this weekend. Wife had plans for me. Going to try to catch up all the work you guys have done tonight

[l-arnold]

Your script works, one at a time.
97-odoo-restart-services
but the goal is getting the -u to apply
so back to it
(is the "openerp-server.pid" calling the same "service" ?

(script lines below)

Stop the services

echo "Restarting services"
echo "Stopping apache, openerp and postgres"
systemctl stop apache2.service
systemctl stop openerp-server.service
systemctl stop postgresql.service

Start them back

echo "Starting postgres, openerp and apache"
systemctl start postgresql.service
systemctl start openerp-server.service
systemctl start apache2.service

[DocCyblade]

What is the -u? I must have missed something!

[l-arnold]

Agree somehow about Apache, but restarting it by itself ?? Didn't seem to work.

the -u is about what to do when the system has "git pulls" run after installation (note above abit w/ reference)
(right below "red" closed marker.
...before any upgrade, always do a backup of your database. Then, do a git pull on every branch, stop your server and start it again with the option -u base in order to update the modules.

[l-arnold]

Can take "apache" out by removing 127.0.0.1 on XML and RPC interfaces in openerp-server.conf then opening port http: 8069 (which Apache otherwise proxies) (not https)

[DocCyblade]

I think know what you are referring to. We are leaving for a it but I'll take a look when we get back.

[DocCyblade]

Maybe we can talk over the phone a bit. Might make things a little clearer to know what needs to be done etc. I can call you or Skype or whatever you perfer.

[l-arnold]

That would work maybe right now. I just opened up the 127.0.0.1 settings and tried to engage the logs also. Still "Service Unavailable" on https and http:8069

(phone dropped for google reasons)

When is best? Now or in a few hours. Was about to go kayak w/ my wife (my turn)

[l-arnold]

I've not been able to get any writing to the log file /var/log/openerp-server.log

Been in a "dummy build" . Probably should come back with a new build.

A few hours from now would be best for me. Does that work? (maybe 9:15 PM) Eastern

[JedMeister]

Have you checked the Apache logs? (/var/log/apache2/error.log)?

Also I'm not sure if this is at all relevant or useful but when I first looked at this appliance (and Odoo in general) I made up a (IMO) more robust init script (see here) which I put in the overlay (overlay/etc/init.d/openerp-server). It's an old sysvinit style one but it works fine in Jessie with systemd (although you either need to use service openerp-server start or /etc/init.d/openerp-server start).

[l-arnold]

@DocCyblade and @JedMeister (I am going to leave Scripting to the two of you if I can. I will say though that the start of the openerp.init came from Carlos (who I have been having a hard time contacting recently). It should work which should work came from his root Git which is here.
https://github.com/CLVsol/tkl-odoo

[l-arnold]

My Old Openerp7 Notes are here on starting the server:

Start File is Created with the following in:
/opt/openerp
(best to edit within Webmin)
#!/bin/bash
./server/openerp-server --xmlrpc-port=40069 --netrpc-port=40070 --addons-path=addons,web/addons $1 $2 $3 $4 $5&

to invoke it I have to use (/bin/bash) below:

su - openerp
/bin/bash start

(start file - basically a bunch of variables that get pulled in
)
#!/bin/bash
/opt/openerp/server/openerp-server --xmlrpc-port=40069 --netrpc-port=40070 --addons-path=addons,web/addons,source/e-commerce-addons,source/openerp-connector,source/openerp-connector-ecommerce,source/openerp-connector-magento,source/openerp-product-attributes,source/sale-wkfl $1 $2 $3 $4 $5&

[DocCyblade]

I'll be available in few hours. Have fun

[l-arnold]

Perfect. We can ping here. Just out the door.

[DocCyblade]

So looking at service script that allows odoo to act as a service. (https://github.com/l-arnold/tkl-nomadic-odoo/blob/kens-dev-wip1/overlay/opt/openerp/openerp.init) there is no -u option. Looking at the documentation I found the -u for odoo.py file (https://www.odoo.com/documentation/8.0/reference/cmdline.html)

[DocCyblade]

I need to head to bed, will be very early morning for me. What time (yours) would be best for us to have a conversation. Thanks

[l-arnold]

I'm guessing that is the Odoo inc version og Odoo.py. (not the one we have buy do not use, I think, in our Firstboot Inithooks)

Good find!

[l-arnold]

I'm back just now.

Prob any time before 8:30 am (MST) my time 10:30 EST.

I will do some tests around odoo.py.
Need to look up again how to call a py gile on the command line.  Prob "  ? "

Sleep well and thank you!

[JedMeister]

Need to look up again how to call a py gile on the command line. Prob " ? "

Do you mean execute/run a py file?
Assuming it has correct shebang (e.g. #!/bin/python) then it should run just like any other script:
./script.py
(assuming that you are in the dir where the script exists)
or
/full/path/to/script.py

You can also launch via python:
python path/to/script.py

[DocCyblade]

I'll be home by 2:15 your time and should be available until 7pm your time.

Regarding the data migration, we could write a helper script that would stop odoo, then interactively select a database then in turn call the scripts to do the upgrade (-u)

I think however I don't think this has to be in the first release.

I propose the first release should contain base install. Try to keep it simple. Remove anything that may not be needed.

Future releases could contain better startup screens to maybe even select what plug ins and add-ons to be included so we can keep the image small, not everyone will want every part of odoo.

Also to keep it smaller, maybe we should remove the .git (this would keep you from doing an update by gitpull but will save space. Again a startup script and maybe a conf console module or screen for added components to odoo would be in order.

Just some thoughts.

[l-arnold]

Sounds all good. if I can activate a manual update I will be comfortable with what we have.

We only have 1 level of git so I don't really think we should take that out.. If we do, we might consider docs to tell people how to put it back in but that is also extra work.

Anyway, need to change computers to test a "script" start. We should be able to get it to go. In the process we might be able add the -u command to the update and hae the "openerp-server-man-conf" perhaps activate it.

I will test.

[l-arnold]

Correcting my earlier comment. We do have "odoo.py" in our firstboot inithooks. It sets the Admin (odoo database creation) password.

There is also odoo.PY (capital PY) in
/opt/openerp/odoo/odoo.PY

Seems to be used "to install Odoo in your environment" but likely more to it than that.
Looking for "call line" I had my hands on a few days ago. Will try to "enable a -u (update) on a fresh database.

*UPDATE
(Not Finding this Code, but just found this Page which is probably better)
http://www.mindissoftware.com/2014/11/23/Odoo-Start-Process/

[l-arnold]

*BREAK THROUGH
Just ran the following and System was UP and Running still at the end of the Loop.
I think we can close this.

I just ran the following and got a running system. I think this means that it "updated" the Database Test.

First Started the System
Second: Defined a Database
Third (as Root) ran the following commands (Note second to last replaces the systemctl start openerp-server.service line)

---

systemctl stop apache2.service
systemctl stop openerp-server.service
systemctl stop postgresql.service

systemctl start postgresql.service
/bin/bash openerp.init start -d test -u test
systemctl start apache2.service

[DocCyblade]

The openerp.init file must pass parameters to the script

[l-arnold]

Details for this has been added to /docs/(textfile)

CAN WE CLOSE the Issue?

[DocCyblade]

So a closer look at the startup file here (https://github.com/l-arnold/tkl-nomadic-odoo/blob/dev/overlay/opt/openerp/openerp.init#L33) shows a call to start-stop-daemon

More info on this command here (http://man7.org/linux/man-pages/man8/start-stop-daemon.8.html)

Looks like command parameters are passed, that is why that worked.

[DocCyblade]

Let me know when you have time for a chat. I am home am should be avail till 7pm your time

[l-arnold]

Now works. (I can call). My office phone otherwise would be good: 303-444-2336
or skype should work: skype name: landis.a

[DocCyblade]

I'll ring you in about 10 min.

[l-arnold]

This Command/Comment is obtuse from a different thread all together on Odoo.com. That said it does have to do with one way to use the (update) parameter -u. Generally these type of paramaters should be able to be applied from the openerp.init (start stop file). (see conf.d/30-main to see openerp.init being initialized in TKLDEV)

sudo -u odoo /opt/odoo/odoo-server/openerp-server -d "DB-NAME" -u all --config=/etc/odoo-server.conf

https://www.odoo.com/forum/help-1/question/how-to-upgrade-database-v8-0-from-saas-versions-56832

[l-arnold]

Details for running the system can be Pointed to in the Documentation set. Generally an Odoo Subject and TKLBAM is restoring insitu.

[l-arnold]

Not sure if some syntax changed in "openerp-server" which was instagated earlier by openerp.init command. However when I try to
UPDATE a Database coming into a Build I am getting an error that looks like this (database name is nomad-dev-wip1-b02)

** root@dev-wip1 openerp/odoo# /bin/bash openerp-server start -u nomad-dev-wip1-b02 -d nomad-dev-wip1-b02
** openerp-server: line 2: import: command not found
** openerp-server: line 8: syntax error: unexpected end of file
** root@dev-wip1 openerp/odoo#

I did just run this command on an old build with openerp.init as documented here. Basically ran "without errors" (above new system with errors). However, on the old system was not running as it was back when we "closed" this issue. Needed to run:
* systemctl start openerp-server.service
to get Odoo running again. Not clear as to root of this.

May reopen issue. Need to test to see if we are having failures in the data. Not seeing any just now.

[JedMeister]

Perhaps your backup includes stuff from an older build which is overwriting updates that have been done to the Odoo appliance since?

I would double check what files are in the backup and use --limits so that it only restores the DB and any essential files. Once you have that working, then take a new backup!

[l-arnold]

This is not really about "TKLBAM". I used ODOO to backup the database on the old system, then Odoo (New Build) to bring that database in. This functionality is quite good.

However, related to the thread here from a few weeks ago, I wanted to see if I could apply the "-u" (UPDATE DATABASE) process so that if there were changes in the newer source code, it would bring those changes to the data-structure if need be.

My new Odoo has the OpenERP Service existing (I believe now) in the /VAR/RUN/openerp-server.pid and we started addressing the file directly and dropped the "openerp.init" pointer (was it a script? I don't think so).

(I will try to test moving the PID File to the original and building a Manual Pointer File to the PID... on the off chance it is looking for other files in the same directory.. Will also look at the old openerp.init but my recollection is that it was just a pointer to the openerp-server (not.pid) file that was in /opt/openerp/ folder. ) All part of the testing/experience building process.

Anyway, those errors showed up when running the commands outlined above. Trying to decipher. Somewhat of an obscure quest. Pretty undocumented but I do understand it is important if you bring a database forward into an updated code base.

[JedMeister]

Ah ok. Sorry I misunderstood.

Re the PID file, it is the init script that calls/uses that. The pid file is just a file that contains the pid (process ID) of the odoo service. The init script uses that to determine whether the server is running or not...

We simply moved the init script (which makes the openerp-server run as a daemon) from the /opt/odoo dir to /etc/init.d (which also removed the need to change permissions and create a symlink etc). It should make no difference to the commands you are running. However I imagine (i.e. just guessing) that the service should NOT be running when you are using the process but I don't know enough about it to be sure...

[l-arnold]

A litle bit of "who's on first". The main thing is the system is running. I will play around and try to get a functional "backend.init" capability somehow. There seems to be quite a bit which can be set. Most of which is in the openerp-server.conf file. I have not found the abiltiy to "update" the database is all.

Could be a big issue if someone runs a Git Pull 6 months later though I expect the Branch to get generally stable. (at least my hope)..

My experience with Joomla and Magento tells me that updating TKL Apps in place is worth the effort so wanting to make sure that works here is also the goal.

[l-arnold]

This is a good outline of the "elements" we want to make sure we have and which we have control of. Here it is outlining what needs to be in place for the Magento Connector to be working properly (should be outlined separately). Irrespective of that, it talks abut how to CONTROL Odoo and have proper Init etc.

OCA/connector-magento#148 (comment)

Well thank you very much for your support, the jobrunner is working like a charm now!

For everyone who does not directly understand the howto, here are the steps that it took me to migrate to the jobrunner:

1. change workers setting in openerp-server.conf:
   my setting is now:
   workers = 10

2. change the startup script in /etc/init.d/odoo:
   add (choose whatever count fits best for you):
   export ODOO_CONNECTOR_CHANNELS=root:3
   export ODOO_CONNECTOR_PORT=8069

and change the start-stop-daemon to:
start-stop-daemon --start --quiet --pidfile $PIDFILE --chuid $USER:$USER --background --make-pidfile --exec $DAEMON -- --config $CONFIG --logfile $LOGFILE --load=web,connector

3. disable "Enqueue Jobs" cron:
   go to Settings->Automation->Scheduled Actions and deactivate Enqueue Jobs

4. update to 3.2.0 of OCA/connector

[l-arnold]

Good documentation on Updating both Odoo Git Files, and updating the data files:

section 4: Updating the database files.

sudo service odoo9-server restart -u all -d YourDatabaseName

[l-arnold]

We must have updated something in the system control. I just updated all the GIT and wanted to be sure did the same with the Database Structure in Odoo.

Above my comment was that we could run:
systemctl stop apache2.service
systemctl stop openerp-server.service
systemctl stop postgresql.service

systemctl start postgresql.service
/bin/bash openerp.init start -d test -u test
systemctl start apache2.service

However, when I do this I get an Stop EXIT In the second to last line:
/bin/bash openerp.init start -d test -u test (replace Test with Database Name)
Not seeing the reason for the exit in the logs. Not seeing whether the Database gets updated or not.

I may be howling at the moon but it seems there should be a "check up" on the dAtabase. Its been several months and there were several changes in code via the GIT.

I realize this is partially an Odoo issue. It is also a question of how we control the Opeerp-Server SERVICE on TKL.

Any thoughts on this?
thanks in advance.

[JedMeister]

I'm not sure what version you are using but v14.1 of the TurnKey appliance changed things a little to make it more reliable.

Whilst in theory systemctl commands should work, IMO it's a bad habit to be in and on Debian/TurnKey I recommend using the service command instead. Maybe in a few years when everything is configured properly for systemd it will be ok, but ATM lots of Debian services don't have proper systemd config setup and all sorts of weird things can happen.

The service command is much more generic and allows the system itself to decide on the best way to start/stop/etc services.

Also using /bin/bash to execute files is another bad habit IMO. IMO if a file should be executed then it should have the execute bit set (i.e. chmod +x /path/to/file) and executed either by changing to the relevant direct and directly executing it, or using the full path. I.e:

cd /path/to
./file

or

/path/to/file

So to update system
:
instead of systemctl stop apache2.service use service apache2 stop
instead of systemctl stop openerp-server.service use service openerp-server stop
instead of systemctl stop postgresql.service use service postgresql stop

(do git pulls here then)

instead of systemctl start postgresql.service use service postgresql start
instead of /bin/bash openerp.init start -d test -u test use /etc/init-d/openerp-server start -d test -u test
instead of systemctl start apache2.service use service apache2 start

(replace "test" with the name of your database)
Not sure if it actually worked but does give a file lock, then file clear in that process.

---

Finally Restart Server for good measure.
(using systemctl start openerp-server.service" does not properly start the service because of the PID file.....