xtreemfs client cross-compiled for Android phones

It took a while, but I have finally managed to cross-compile XTreemFS client for Android!

First, there are two posts that I’ve tried posting on the XtreemFS group, but I think they were marked as spam and deleted automatically by the system. They are below:

Post 1:

“I’ve just successfully mounted the volume on a Galaxy S3 🙂 (rooted stock rom) YEEEEY.

My target is Android … I have managed to cross compile it and mount the volume!

root@android:/system/xbin/lib # ./mount.xtreemfs 192.168.0.101/MyVolume /mnt/xtreemfs/
[ E | 2/27 19:19:14.409 | 400 ] failed to retrieve passwd entry for uid: 0
[ E | 2/27 19:19:14.411 | 400 ] failed to retrieve group entry for gid: 0
root@android:/system/xbin/lib # cd /mnt/xtreemfs/
root@android:/mnt/xtreemfs # ls
20120606_080341.jpg

All 4 executables were linked statically (After many many many changes ….) against ucLibc and this is the size:

root@android:/system/xbin/lib # ls -al *.xtreemfs

-rwxr-xr-x root root 14438887 2013-02-27 19:04 lsfs.xtreemfs
-rwxr-xr-x root root 14824770 2013-02-27 19:04 mkfs.xtreemfs
-rwxr-xr-x root root 15650546 2013-02-27 19:04 mount.xtreemfs
-rwxr-xr-x root root 14781776 2013-02-27 19:04 rmfs.xtreemfs


They are not stripped – I guess that would take some Mb off of each one.

Now, the problem:

I decided to skip over LIBATTR (another hurdle to compile, I wanted quicker results) and since it is optional on Solaris, I thought maybe is not critical on Linux either – hehe, I don’t even know what role it plays. But I will try to get a build with that included also. So it is currently not, but I’ll explain the behavior:

I have 3 clients running, 2 on Windows 7 64bit, 1 Android phone. One of the Windows machines is running a minimal setup of the server – no security, no auth – just “in house” testing.

Whatever I upload from Android, it takes about 1 minute to be visible to the Windows machines. However, the file is not readable (I mean I cannot copy it, open it, nothing). Media Player says “File not found”, Total Commander says “Access denied”.
Whatever I upload from the Windows machines, is visible in about 30 seconds to the other Windows client and content is correct, BUT … they are not readable on Android.

These are the contents of the volume, as seen from the Android mount.

root@android:/mnt/xtreemfs # ls -l

-rw------- u0_a55534 u0_a55534 355624 2012-06-06 06:30 20120606_080341.jpg
-rw-rw-r-- root root 5006550 2012-06-09 14:21 20120609_162139.mp4
-rw------- u0_a55534 u0_a55534 226447 2013-02-27 19:33 blender.png
-rw------- u0_a55534 u0_a55534 241874 2013-02-27 19:35 showthread2.php


The mp4 was uploaded using Android, the others were uploaded using Windows clients.

If I try to change the rights as root, I get “Operation not permitted”. So it could be libattr – but I will see when I’ll give it another try with libattr too …

strace chmod 777 showthread2.php on Android shows:

mprotect(0x4007a000, 4096, PROT_READ) = 0
chmod("showthread2.php", 0777) = -1 EPERM (Operation not permitted)
mprotect(0x4007a000, 4096, PROT_READ|PROT_WRITE) = 0

Thank you for feedback,”

Post 2: – this is a second attempt to reformulate Post 1 above, hoping that it will get through.

“Hello Michael and thank you for the clarification with the message.
It was about finally being able to build the client for ARM and executed the code successfully on my S3 Android phone!

To clarify the fuse / cmake point – I am using buildroot as a toolchain (so that I can have a more complete libc available). In this setup, I have to statically link everything together (libc included) since the ld.loader existing on the phone won’t be able to load the .so libraries (unless I dig waaaay to deep into the issue, I prefer to statically link everything together).

CMakefile.txt, the way it currently is – it will search for the .so version of the libraries and pass them to the linker, but I had to change it so that the static versions are used. CMakefile does provide for an alternate location for boost library (BOOST_ROOT variable) to allow the user to specify an alternate boost to be used, not necessarily the one on the system. However, currently there’s no provision for an alternate Fuse library to be specified like this. But since I needed to change in many places to build statically, this was better in fact 🙂

Anyway, back to the results – here’s the size of the binary files:

root@android:/system/xbin/lib # ls -l *.xtreemfs
-rwxr-xr-x root root 14438887 2013-02-27 19:04 lsfs.xtreemfs
-rwxr-xr-x root root 14824770 2013-02-27 19:04 mkfs.xtreemfs
-rwxr-xr-x root root 15650546 2013-02-27 19:04 mount.xtreemfs
-rwxr-xr-x root root 14781776 2013-02-27 19:04 rmfs.xtreemfs

And I was also able to mount the volume on my mobile phone!

root@android:/system/xbin/lib # mkdir /mnt/xtreemfs
root@android:/system/xbin/lib # ./mount.xtreemfs 192.168.0.101/MyVolume /mnt/xtreemfs
[ E | 2/28 05:38:54.605 | 400 ] failed to retrieve passwd entry for uid: 0
[ E | 2/28 05:38:54.607 | 400 ] failed to retrieve group entry for gid: 0
root@android:/system/xbin/lib # ls -l /mnt/xtreemfs
-rw------- u0_a55534 u0_a55534 355624 2012-06-06 06:30 20120606_080341.jpg
-rw-rw-r-- root root 5006550 2012-06-09 14:21 20120609_162139.mp4
-rw------- u0_a55534 u0_a55534 226447 2013-02-27 19:33 blender.png
-rw------- u0_a55534 u0_a55534 241874 2013-02-27 19:35 showthread2.php
root@android:/system/xbin/lib #

The good part is that the volume contents are read, and I can also write ! My setup is a very basic setup (two Windows 7 machines 64bit, both act as client but only one has a minimal server running). And the S3 Android phone acting as client. There’s no security, no encryption for now, just to see how thing work.

Above, the .mp4 file was uploaded from the phone, the other files from the W7 machines.

The trouble is that whatever I upload from the phone is not readable by the W7 clients (I get errors like “File not found”, or “Access denied”) and the same is true on the mobile for the files that I upload from the W7 machines.
W7 client can successfully read the content uploaded by the other W7 client, but the phone cannot.

root@android:/system/xbin/lib # cat /mnt/xtreemfs/showthread2.php
sh: cat: /mnt/xtreemfs/showthread2.php: Permission denied
1|root@android:/system/xbin/lib # chmod 777 /mnt/xtreemfs/showthread2.php
Unable to chmod /mnt/xtreemfs/showthread2.php: Operation not permitted

However, there one thing missing for now: libattr – it was another hurdle in getting the whole thing built and since it is not needed for Solaris, I thought that maybe it is not critical for Linux / Android either! So I commented it out – I will give it another shot with it included.
I never used libattr in the past 😀 so I have no ideea what it does and not having it can explain the behavior.
Also, my kernel comes with fuse built in (I grepped kallsyms and see stuff like fuse_fs_type and fuse_mutex and so on) so I did not have to do this step either.

Thank you for your feedback,

Moving forward:

As noticed above, there are also more dependencies which needed to be compiled:
a) attr (which needs libintl.h)
b) gettext (which provides libintl.h)

Luckily, gettext comes with build root, so the only thing needed is to cross-compile attr.

Before getting to this, I will post below another post that I posted to the mailing list that describes the hurdles of cross-compiling:

“Hello,

I’ve been trying to cross-compile the client only. Although I do have some experience with cross-compiling, I have to say that the build system is very complex:
a) make is needed to kickstart the things and compile the thirdparty libs then hand it of to cmake for the good parts.
First hurdle are the thirdparty libs, since they fail because protobuf has some unit tests which generate an error because the binary files are not executable on the host. I had to comment out parts of the global Makefile and manually compile the thirdparty libs. I got the .a files, I didn’t care about the unit tests for now.
b) then make hands over the control to cmake, which is needed for the client. It too needed to be told to generate a cross-compiled version of whatever it was going to build. First thing encountered, boost dependency (66mb of a tar.gz that needs to be cross-compiled too).
c) but boost has its own way of being built – you bootstrap it, then you configure the project definition, and so on. This is the third step on having to compile Xtreemfs client; there are lots of errors generated in the process (such as untold dependency for python2.6-dev which needed to be installed beforehand). There were still errors there, but I got the libs required so ok. Moving on.
b) then I got to fuse at which I stopped …

So there are 3 build systems needed if you need to compile everything from scrap. I don’t want to think how big the binary client will end up to be if I statically link everything together.

I understand that it might have taken a way longer time to use just autotools (or equivalent) but it would help a lot to one build system and use standard C++ without all the huge dependency.

So the question is – is boost really needed for the client ?

Thank you,”

The answer I got is that yes, everything is needed.

The problem was not related to libattr but with the access policy. By default, when creating an xtreemfs Volume, the access policy is POSIX – meaning that user/group names are used. Unless you have the exact user/group on all machines (Android / Windows) the files will be uploaded, but only the clients having the same user/group will be able to read them.

Digging a bit through the manuals (chapter 2.3.4) as well as the source code, it became apparent that POSIX won’t work with Android / Windows combination because:
a) username / group names are missing on an Android phone (no /etc/passwd or /etc/group files)
b) I would have to create another username for Windows and find a way how to change the group names too.

I’ve created the Volume with the Windows client using the command below (There are warnings related to the NULL access policy, please search manuals if you don’t know what risks it has).

C:\Program Files (x86)\XtreemFS Windows Client>mkfs.xtreemfs.exe -u root -g root -a NULL 192.168.0.101/MyVolume
Trying to create the volume: 192.168.0.101/MyVolume

Successfully created volume "MyVolume" at MRC: 192.168.0.101/MyVolume


And I was able to upload a photo from the phone and see it on the Windows machine about a minute later 🙂