Memcache - sharing variables

More
23 Jan 2012 00:08 #17029 by dab77
Hi i just discovered python-memcache, and i've found it so simple and useful that i think it can be useful to others.
I have one gladevcp window with my buttons and LEDs, one vismach window and another gladevcp window with other infos on my machine.
I needed a way to make those three programs comunicate and i've found this memcache, which reserve some Mb of memory and let you save any datas in a very simple way.

just put:

import memcache
mc = memcache.Client()

into any python file you want to intercomunicate and then to write:
mc.set("value_name","value")

and to read:
my_value = mc.get("value_name")

Into ubuntu, you need just to install:
sudo apt-get install memcached python-memcache

memcached is the daemon that set th11211 port and reserve memory.
if interested you can find other infos here: blog.isotoma.com/2009/09/memcache-in-2-minutes/

Wish it's helpful
Davide.

Please Log in or Create an account to join the conversation.

More
23 Jan 2012 07:13 #17035 by ArcEye
Hi

Interesting.

Looks like it is a socket server running on localhost:11201

Co-incidentally I am presently writing a socket server in C to interact with userspace modules and store the values of HAL pins in realtime.

I don't use python, but will have a look at it to see if there are any ideas in the mechanism of it that I can 'borrow':cheer:

regards

Please Log in or Create an account to join the conversation.

More
23 Jan 2012 10:03 #17037 by mhaberler
As you noted, there is a lack of a coherent, late binding IPC mechanism beyond HAL and NML, and this issue comes up in many corners of linuxcnc, not just between Python scripts. Also, persistent state (state retained across linuxcnc invocations) is an issue as this is currently, well, 'unstructured', so it isnt just a question of IPC.

I am addressing this issue as I work on a rewrite of the interpreter and task (mostly its structure, not its contents). I have decided to use redis as such an IPC mechanism, and have partially integrated it.

While likely will not be something tangible for others to try before summer, I encourage anybody seriously looking into such a mechanism install redis (easy) and use redis-py (easy to use) as a python binding. It can only help if we collectively gain some experience with it.

http://http://redis.io/
github.com/andymccurdy/redis-py

- Michael

Please Log in or Create an account to join the conversation.

More
23 Jan 2012 14:05 #17050 by dab77
Michael, you say right.
going to study this redis, which i never heard before.

p.s. just an add-on with memcache, you can start a client from every programming language, so into C you can recall the vars saved from Python or whatelse.

Please Log in or Create an account to join the conversation.

More
23 Jan 2012 15:59 - 23 Jan 2012 16:11 #17055 by ArcEye
Hi Michael

It can only help if we collectively gain some experience with it.

This looks very promising, a key based database server would solve some of the problems of socket servers that I have discovered.

The primary one is that connecting remotely reliably, necessitates data be serialised to byte level, otherwise there is no way to know the data sizes used on the remote machine ( is int 16 bit or 32? what padding is used, etc) so you cannot pass structs etc.
All the converting values to strings and back again is quite unwieldly and requires rigid message formats.

I will study this to see if it prevents me having to re-invent the wheel!

regards
Last edit: 23 Jan 2012 16:11 by ArcEye.

Please Log in or Create an account to join the conversation.

More
24 Jan 2012 11:03 - 24 Jan 2012 11:10 #17094 by ArcEye
For those who prefer to program in C, have found these libraries for redis which implement most of the client commands

code.google.com/p/credis/

Having a play with them now, the example from the wiki will throw a seg fault unless you spot that it uses an arbitrary port address of 6789 despite the default for redis being 6379.

Change that and it works fine so far

regards
Last edit: 24 Jan 2012 11:10 by ArcEye. Reason: add info

Please Log in or Create an account to join the conversation.

More
25 Jan 2012 00:09 #17106 by mhaberler
You pulled an dud

redis.io/clients clearly marks suggested clients, and credis isnt one of them

stick to the bindings which have a star marked on this page, those are actually used, and I still have to find a problem with them

My suggestion is: for C, use hiredis, and for Python: redis-py

- Michael

Please Log in or Create an account to join the conversation.

More
25 Jan 2012 12:41 - 25 Jan 2012 12:47 #17122 by ArcEye
Calling credis a dud is a little unkind, it works as it claims, even if the documentation consists solely of the credis.h file and a single wiki page example that does not work (as previously covered) so far as I can find.

However having looked a hiredis, I do like the printf style functions, keeps the input nearer a command line entry and handles some of the string formatting you would have to do before the function call anyway.

I only have one afternoon invested in credis, think I can probably switch to hiredis and port my code so far in about 10 minutes!

regards
Last edit: 25 Jan 2012 12:47 by ArcEye.

Please Log in or Create an account to join the conversation.

More
25 Jan 2012 17:28 #17129 by mhaberler
yes, my wording wasnt appropriate, sorry. I'll refrain from posting after beer #2 :S

I should write down my current plans for restructuring LinuxCNC in a wiki page, to outline how redis fits in.

the status currently is:
- the following components already have basic redis capability: task, interpreter, TkInter Python applications like Axis, GladeVCP (in fact all PyGTK or GTK applications like Touchy), any userspace C or Python HAL component.
- they can both use the redis set/get and derived mechanisms to export/retrieve global variables
- they also can use the publish/subscribe mechanism to declare interest in what is called a 'channel', and get notified asynchronously when a channel-related event happens, for instance, when a variable changes value (this is why the async support is needed).

As mentioned, I use hiredis for C, and redis-py for Python for synchronous I/O. For publish/subscribe I currently use Twisted, which is the fasted path to get this to work on the Python side, but there are some minor issues left with TkInter apps.

The open issues are:
- design a redis data model which includes the tool table, and possibly other state components in an extensible way
- implement this without breaking backwards compatibility (it will be a lot of work until all user interfaces are adapted, and I dont know others than Axis and Touchy very well)
- export/import to/from external files like tooltable

I'd be interested to discuss any plans when accessing interpreter and task state.

- Michael

Please Log in or Create an account to join the conversation.

More
25 Jan 2012 18:32 #17131 by BigJohnT
mhaberler wrote:

I'll refrain from posting after beer #2 :S

- Michael


Hmmm, I might need to set my posting enable breath analyzer a bit lower. :)

John

Please Log in or Create an account to join the conversation.

Time to create page: 0.095 seconds
Powered by Kunena Forum