1 Welcome to EQ2430/EQ2440 Android Lecture March 20, 2014 Per Zetterberg School of Electrical Engineering
1
Welcome to
EQ2430/EQ2440 Android Lecture
March 20, 2014
Per Zetterberg
School of Electrical Engineering
2
The project course goes Android!
• Display : plots and text.• Microphone.• Loudspeaker.• Magnetometer measurements.• WiFi RSSI measurements.• Send data using WiFi.• Accelerometers.• Touch screen.• Camera
Features used in our projects:
3
FrameWork• FrameWork is a java project (program) in
Eclipse.• FrameWork is a Android 4.2.1 project that
runs (at least) Samsung S3. • FrameWork is a generic skeleton which can
be used as the starting point for many student projects (including yours!).
• FrameWork has an object StudentCode which is where you implement your prototype.
• Application name: StudentFrameWork (Change this)
• Creators: Martin Ohlsson and Per Zetterberg
4
FrameWork: User interface
PlotView:
Graphs and/orCamera picture
TextView:
Textual output.
5
Downloading FrameWork
Go to
EQ2440 web (KTH social) => Android support
Download: FrameWork.zip
Unzip and move the code to workspace directory.
Do “file->import->”existing project into workspace”-
>browse to FrameWork (the directory)
6
StudentCode (i.e your code)Parameters:useSensors, loggingOn, logSound, useCameraGUI, useMessaging, messageServer, processInterval, introText, messageGroups, userInputString, StudentMessage (class), userInputString
useSensors (bitfield): “xor” the followingGPS,CAMERA,CAMERA_RGB,SOUND_OUT,SOUND_IN,WIFI_SCAN,ACCELEROMETER,MAGNETIC_FIELD,PROXIMITY,TIME_SYNC, PROXIMITY,LIGHT,SOUND_IN,CAMERA, GYROSCOPE
7
Called by StudentCode 1(3)sound_out(short[] buffer, int length): Send data to DAC=>loudspeaker.
message_out(StudentMessage message): Send message to all the other phones.
message_out(StudentMessage message, String groupIdString): Send message to the phones in group defined by “groupIdString.
streaming_buffer_out(short[] buffer, int length, String groupIdString): Send buffer of int16 to the phones in group defined by “groupIdString”.
p_streaming_buffer_out(short[] buffer, int length, String groupIdString): Send buffer of int16 to the phones in group defined by “groupIdString”.
8
Called by StudentCode 2(3)set_output_text(String text): Set the text too be displayed in TextView.
plot_camera_image(Canvas plotCanvas, byte[] image, int imageWidth,int imageHeight, int width, int height): Display a camera image.
plot_camera_image_rgb(…): Display color …
write_string_on_logfile(String): No ”\n” or ”;”.
load_sound_file(String fileName) + play_sound_file(int loop): Repeatedly plays the wav-file specified by fileName. No timing gitter (loop=-1 means forever)stop_sound_file(): Stops the above.
9
Called by StudentCode 3(3)
save_rgb_frame_on_file(String filename,byte[] frame,int width,int height,boolean append) : Saves an RGB picture (=frame) on file.
public byte[] read_rgb_frame_from_file(int image_number, String filename) : loads an RGB picture (=frame) from file.
public byte[] read_data_from_file(String filename) : Reads binary data from file.
10
Defined by StudentCode: Called by FrameWork 1(2)init(): Do all your initialization.start(): Called when menu button ”start” is pressed.stop (): Called when menu button ”stop” is pressed.message_in(…) Received message from other phone.streaming_buffer_in(…) Buffer of 16bit ints.gps(…) …… position estimatemagnetic_field(…) …… magnetic field measurementlight(…) : …. Light intensity (lux) event (trigger when changed)proximity(…): Proximity state event (trigger when changed)accelerometer(…) accelerometer measurementprocess(), Periodic processing with interval set by
processInterval parameter. The processInterval parameter can be changed during runtime.
You fill these functions with code!
11
Defined by StudentCode: Called by FrameWork 2(2)gyroscope(…) gyroscope measurementsound_in(…) … sound input sampleswifi_ap(…) ... signal strength of surrounding WiFi APsscreen_touched(…) position eventstreaming_sound_in(…) buffer of samples (int16) received from other phonecamera_image(…) image received from camera (black and white)camera_image_rgb() image received from camera (color)plot_data(): Screen update (make plots with Canvas).list_ips(): Return a list (in string format) of the IP:s of the phones in your messaging group.stringFromUser(): Called when the user provides input.
All are inputs exceptplot_data!
You fill these functions with code!
12
Your code structure
public void init() { … set parameters initialize}
public message_in () { … process … save}
public void process() { … e.g heavy calculations}
camera_image_rgb() { … save latest imagenew_image=true;
plot_data(Canvas plotCanvas, {… present results}
Data structuresalgorithm state
13
The GUI
“Greeting message”
introText
Start FrameWork App-> calls init()
Press menu button
Start() is calledSensors switched +on etc.
Stop
Press Start
set_output_text(String text): Set the text too be displayed in TextView.
plot_data(Canvas plotCanvas,…)
Press menu + Stop
stop() is called
Start Kill String
14
Get a string from the user 1(2)
Menu if userInputString=false:
Start Kill
Start Kill String
Menu if userInputString=true:
Pressing String gives:
Message
userInputStringMenuItem=“String”
Title
OK Cancel
userInputStringTitle=“Title”
userInputStringMessage=“Message”
15
Get a string from the user 2(2)
void stringFromUser(String user_input){ }
Title
OK Cancel
Orange User types ”orange”
User presses OK
The input providedis sent to the FunctionstringFromUser insideStudentCode.java
16
Browse for filename
To use it, somewhere inside StudentCode.java do:
browseForFile();
This opens a dialog where you can browse for a file (quite intuitive). When the user selects a file and presses ”Done” the function stringFromBrowseForFile is called back. You can capture the filename inside it.
stringFromBrowseForFile(String filename)
Only the variable filename is set in filename not the path. However, we will only use the sdcard directory.
17
Threads (internals)“Parent thread”: Indirectly calls init().studentProcessThread: Calls process(). ipListenerThread: Calls message_in()guiTriggerThread: Renders what you provide with set_output_text() and plot_data(). t1: Calls sound_in(…)wifiScanThread: calls wifi_ap(…)ipSoundListenerThread: calls streaming_buffer_inFrameWork implements OnRecordPositionUpdateListener:
Calls gps()FrameWork implements SensorEventListeneraccelerometer(), magnetic_field(), light(), proximity()FrameWork extends Activity: sreen_touched()streaming_sound_in(…) buffer of samples (int16) sent from other phone
18
How communication between phones works 1(2)
• A phone (or a Linux computer) acts as a server (the address of which is defined by messageServer).
• When using a phone as server, that phone is starts FrameWork first.
• When FrameWork starts (on the other phones), it contacts the server and asks for the ip adress of one of the other phones in the project (a project is defined by the projectName string e.g. ”yellow”).
• The phone then contacts that phone and gets a list of all other phones in the project.
• When one phone calls message_out(message) the message is sent to all other phones in the project.
• The other phones receives the message through a call of message_in (”callback”).
• The message is of type StudentMessage and contains the fields that you decide.
• Debugging hint: set_output_text(set_list_ips()) lists current configuration.
19
How communication between phones work 2(2)• Second form of message_out: message_out(message, groupIdString)
• Where groupIdString defines a group of users e.g. ”data_receivers” which should receive the message.
• Which groups exist ? You define the groups with the ”messageGroups” parameters.
• When FrameWork starts, it asks the user which group the phone belongs to.
• When sending buffers of int16 use the following version: streaming_buffer_out(buffer,length, groupIdString)
• This data is received through: streaming_buffer_in( buffer, int length, senderId)
20
How to use communication between phones 1(2)Set useMessaging = true;
If you need groups:•String temp[] = {"MICROPHONE","SPEAKER"};•messageGroups = temp;Otherwise:•messageGroups = null;
In both cases:•messageServerType = LINUX_MESSAGE_SERVER; •messageServer = "192.168.0.68";•Define StudentMessage (defined in StudentMessage.java)
Which message group do this phone belong to ?:messageGroups[myGroupID]
21
How to use communication between phones 2(2)public class StudentMessage extends StudentMessageBase {// Add your message fields here// do not use primitive types that begins with lower case letters Float dummy=0.0F; /* Example of student message data */ Boolean is_sync_message; Boolean is_distance_message; Float value1; Float value2; Float value3; Float value4;}
Same definition in all phones!
message_out(my_message,SPEAKER”);In transmitter
public void message_in(StudentMessage message){If (sync_message) correlation=value1;…
In receiver(SPEAKER)
22
Buffers of int16
p_streaming_buffer_out(buffer,length, groupIdString)
In transmitter:
In receiver:
streaming_buffer_in(short[] buffer, int length, int senderId)
New version:
streaming_buffer_out(buffer,length, groupIdString)
As streaming_buffer_out but does not start a TCP/IP connection each time.Also only one phone per group allowed.
23
Logging on file 1(2)
• loggingOn = true;• Logs measurements from all active sensor except
camera • … and you own debug messages:
write_string_on_logfile(“correlation=“+correlation)• Creates a measurement file called sensorlogYYYYMMDDHHMM.csv on the SDCARD.
The measurement file is ASCI with measurents in clear text.
How retrieve file:method1: Copy using windows exlporer.methods: Using our matlab functions.
24
Logging on file 2(2): matlab utilities
• Copy file to PC-harddisc (matlab command)copy_file_from_sdcard_to_working_directory.m • Obtain the data from the log-file into matlab:get_log_data_from_FrameWork.m• Extract data from some certain sensors:
• extract_acceloremeter_from_log_data.m,• extract_sound_from_log_data.m, • extract_magnetic_from_log_data.m
• Other sensors:• Do it yourself.
Problem: SOUND_IN fluds measurement log.
Solution: logSound=false;
25
State-machine and functionspublic void process() {{
if (!buffer_available())return;
switch(State) {case START:
res1=func1(buf,arg2); res2=func2(signal1);
State=WAITIING;break;
case WAITING:sync=func3(buf,res2);if (sync) {State=RUNNING; transmit(signal2)}
else {State=WAITING; transmit(signal1)};break;
case RUNNING:.....
}write_string_on_logfile("State="+State);
}
public void message_in(StudentMessage message){
switch(State) {case WAITING:....
}
Functions that can be tested against matlab implementation using harness
26
Test harnessPurpose: Test implementation of math-intensive functions/algorithms in an isolated and repeatable way.Method:Run your matlab- and java- (with android environment) -implementations side by side with exactly the same input .How?•Matlab produces data, and pushes it on the sdcard•FrameWork includes a function called ”test_harness()”.•Inside this function
the data can be read from the sdcard,
the function to be tested is called,and the results are saved on the sdcard.
•Matlab then pulls the data from sdcard.•Finally, the results from the matlab- and java-implementation are compared.What do I need to do ?The example test_harness_example.m shows how to automate the process. This code can be used as a starting point, however, you need to customize the code to the function you are going to test.
All, non-trivial functions should be tested like this.
Between changes in the code do:Save and press run. Then run the harness again.
27
Using sound (input)
time4096 samples 4096 samples 4096 samples
useSensors = SOUND_IN | SOUND_OUT;
sampleRate = 44100;
sound_in(long time, short[] samples, int length)
sound_in(long time,…
28
Sound output 1(2)
sound_out(short[] buffer, int length)
sound_out(short[] buffer, int length)
sound_out(short[] buffer, int length)
-2^15,2^15-1
29
Sound output 2(2)
load_sound_file("bing.wav"); play_sound_file(-1);stop_sound_file();
wavwrite(y,44100,16,’bing.wav’);copy_file_from_working_directory_to_sdcard(‘bing.wav’);
Only in init(), process() and stop()
Constant gaps
30
Create sound in matlab and play it on phone
In Matlab•[y,Fs] = wavread('C:\Downloads\lga.wav');•create_file_of_shorts('lga.dat', y(20001:1e6,1)*1000 );•copy_file_from_working_directory_to_sdcard('lga.dat' )
On the Phone:
See lines in in StudentCode.java with the word ”sound”
In matlab sound: [-1,1]. In android FrameWork: [-2^15,2^15-1]
31
RGB Camera: Image formatvoid camera_image_rgb(byte[] image, int width, int height)
0
1
3
image={r0,g0,b0,r1,g1,b1,r2,g2,b2,…}
w
1w
1wh
Byte 0-255
1w 12 w
32
RGB Camera: File format
File format (”.rec”):
width height r0,g0,b0,r1,g1,b1,….
Frame
Frame 1 Frame 2 Frame N
Always stored in /sdram/recordings
Filename (example): video201303181626.rec
33
RGB_Camera: GUI
Alt1: useSensors = CAMERA_RGB; useCameraGUI=true;
Start Rec Play Stop
void camera_image_rgb(byte[] image, int width, int height)
Log to file: /recordings/videoyyyyMMddHHmm.rec
Browse for file and playback: Calls camera_image_rgb(byte [] image, int width, int height)
Stop recording
34
RGB_Camera:No GUI
Alt1: useSensors = CAMERA_RGB; useCameraGUI=false;
Camera runs continuously.
void camera_image_rgb(byte[] image, int width, int height)
….is called….
save pictures to file using call:
save_rgb_frame_on_file(String filename,byte[] frame,int width,int height,boolean append)
35
Load RGB picture from file
int read_rgb_frame_width_from_file(int image_number, String filename)
int read_rgb_frame_height_from_file(int image_number, String filename)
byte[] read_rgb_frame_from_file(int image_number, String filename)
36
Display picture (example)
plot_data(Canvas plotCanvas, int width, int height) {
if((latestRGBImage != null) && ((useSensors & CAMERA_RGB) == CAMERA_RGB)){
plot_camera_image_rgb(plotCanvas,latestRGBImage,ImageWidth,imageHeight,width,height);
};};
Updates≈10Hz
37
Pictures Matlab interface
list=list_video_recordings_on_sdcard(time0)
copy_video_recording_file_from_sdcard_to_working_directory(filename)
rgb_video= extract_video_recording_from_file(filename);
image(rgb_video.frames{1}.rgb)
create_video( filename, rgb_video )
rgb_video.frames{1}.rgb A 768x432x3 matrix(:,:,1)=red (:,:,2)=green (:,:,3)=blue
38
Display pictures another way
void plot_data(Canvas plotCanvas, int width, int height) {
plotCanvas.drawLine(… plotCanvas.drawRect(……}
http://developer.android.com/reference/android/graphics/Canvas.html
More info:
39
ZXING: QR decoder
“ZXing (pronounced "zebra crossing") is an open-source, multi-format 1D/2D barcode image processing library implemented in Java”
To test:useSensor=CAMERA_RGB; uncomment code in StudentCode.java between ”zxing start color” and ”zxing stop color”
Direct camera towards QR code
When address found result is returned in: res.getText()
Source code located under ExternalSrc.com.google.zxing
40
Give your App a unique identity• Right click on project folder (Top level folder named
FrameWork) within the Package Explorer window• Select Android Tools/Rename Application Package• Add a unique name after FrameWork like:
se.kth.android.FrameWork.GroupRed2013• Press Finish in the Refactoring window• Confirm update of Launch Configurations• Voila, you now have a separately installable version of
your app!
41
Phone configurationSamsung S3 (original android version)”Settings” -> ”developer options” -> tick ”USB debugging””Settings” -> ”Security and lock screen” -> tick ”Unknown sources”
Android versions >= 4.2Goto Settings -> About Phone. -> Tap on “Build Number” repeatedly (7 times) -> back -> as above.
Eduroam (both):http://www.lan.kth.se/eduroam/phones/phones_mobile_guide.html
42
Assignments1. Record accelerometer data and save it a logfile. Do recordings
from the elevator. Can you see detect when the elevator starts and stops from the measurement? Spend a day on trying to make a “floor level display app from this”.
2. Plot a car using “draw” functions and the canvas.3. Switch between a seeing the car from left and right side every
second.4. Record a sound and play it with two echoes. The sound should
be read continuously like a reverb i.e. not record->pause->play.5. Record sound on one phone and play and stream it to another.
Buffering?6. Record sound from two phones. Add the sound from the two
phones and play it on a third phone. Buffering?7. Make a test-harness for a non-trivial algorithm which is relevant
for your project.
Deadline : Mid-term evaluation.
43
Hints• Android software developers kit:
http://developer.android.com/sdk/index.html• Killing tasks. Download and install “Advanced Task
Killer Free (ATKF)”. To kill e.g. “StudentFrameWork” open ATFK press “StudentFrameWork” until a new menu appears where you can select “Kill”
• Error message: “ActivityManager: Warning: Activity not started, its current task has been brought to the front” => Change a single line in the source files code and try again.
• Error message:FrameWork_empty_130319] Re-installation failed due to different application signatures => Do “abd uninstall se.kth.android.FrameWork” from command line prompt on PC.
• Install Audalyzer to be able to view in the input sound.