You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 43ef81a
Browse filesBrowse the repository at this point in the historyBrowse files
# Add ADB driver support for Android VMs
## Description
This PR adds Android Debug Bridge (ADB) support to the VM Resource
Handler so FIREWHEEL can manage Android emulator guests without
requiring the QEMU Guest Agent.
It includes the new `ADBDriver`, VM Resource Handler and minimega API
updates needed to discover and use Android-specific connection metadata,
and the supporting `adbutils` dependency. It also adds Android system
documentation and tutorials for setup, automation, and analysis.
## Type of Change
Please select the type of change your pull request introduces:
- [ ] Bugfix
- [x] New feature
- [x] Documentation update
- [ ] Other (please describe):
## Checklist
- [x] This PR conforms to the process detailed in the [Contributing
Guide](https://sandialabs.github.io/firewheel/developer/contributing.html).
- [x] I have included no proprietary/sensitive information in my code.
- [x] I have performed a self-review of my code.
- [x] I have commented my code, particularly in hard-to-understand
areas.
- [x] I have made corresponding changes to the documentation.
- [x] My changes generate no new warnings.
- [x] I have tested my code.
## Additional Notes
- `adbutils>=2.12.0` is added as a new dependency for Android guest
communication.
---------
Signed-off-by: gregjacobus <30842019+gregjacobus@users.noreply.github.com>
Signed-off-by: Steven Elliott <6461548+sdelliot@users.noreply.github.com>
Signed-off-by: Mitch Negus <21086604+mitchnegus@users.noreply.github.com>
Co-authored-by: Steven Elliott <6461548+sdelliot@users.noreply.github.com>
Co-authored-by: Steven Elliott <selliot@sandia.gov>
Co-authored-by: Mitch Negus <21086604+mitchnegus@users.noreply.github.com>
Copy file name to clipboardExpand all lines: docs/source/cli/commands.rst
+1-2Lines changed: 1 addition & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -42,8 +42,7 @@ config edit
42
42
43
43
usage: firewheel config edit [-e EDITOR]
44
44
45
-
Edit the FIREWHEEL configuration with a text editor. The user must set either the VISUAL or EDITOR
46
-
environment variable or use the provided flag to override these environment variables.
45
+
Edit the FIREWHEEL configuration with a text editor. The user must set either the VISUAL or EDITOR environment variable or use the provided flag to override these environment variables.
Copy file name to clipboardExpand all lines: docs/source/system/architecture.rst
+3-2Lines changed: 3 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,7 +17,7 @@ To accomplish this, FIREWHEEL's *Control* system interprets user defined experim
17
17
FIREWHEEL Software Architecture
18
18
19
19
FIREWHEEL's software architecture, as shown in :numref:`Figure %s <fw_sw_arch>`, consists of a Command Line Interface (CLI), Experiment Models, and three other major components (*Control*, minimega, and the *VM Resource Manager*) that convert experiment models into running experiments, and manage FIREWHEEL's VM deployment, virtual network and VM provisioning, and run time execution capabilities.
20
-
These FIREWHEEL software components are executed within a Python Virtual Environment, which we will refer to as ``fwpy``, and interact with the Hardware Virtualization components, QEMU/KVM and Open vSwitch (OVS), to instantiate experiment models as configured VMs communicating via virtual networks.
20
+
These FIREWHEEL software components are executed within a Python Virtual Environment, which we will refer to as ``fwpy``, and interact with hardware virtualization and networking components, including QEMU/KVM, the Android Emulator when Android VMs are used, and Open vSwitch (OVS), to instantiate experiment models as configured VMs communicating via virtual networks.
21
21
We'll next discuss each of FIREWHEEL's major software components.
`minimega <https://www.sandia.gov/minimega/>`__, is the name of the virtualization management component that's currently available with FIREWHEEL. minimega will instantiate an experiment using QEMU/KVM for VMs, and OVS networks.
106
+
`minimega <https://www.sandia.gov/minimega/>`__, is the name of the virtualization management component that's currently available with FIREWHEEL.
107
+
minimega will instantiate an experiment using supported VM backends, such as QEMU/KVM for conventional VMs and native Android emulator support for Android VMs, and OVS networks.
107
108
minimega receives an experiment graph from FIREWHEEL's *Control* system, then determines and sends the appropriate set of instructions to the QEMU/KVM hypervisor and OVS virtual networking systems that are needed to instantiate the experiment model as an emulated computer network.
108
109
At this stage the experiment network's topology has been fully instantiated on the virtualization platform, but has yet to have any of its scheduled actions applied to it -- which may further configure the compute systems and network devices as needed before beginning the intended cyber experiment.
The `Android Debug Bridge (ADB) <https://developer.android.com/tools/adb>`_ driver enables the VM Resource Handler to communicate with Android guests.
7
+
It is selected using the VM Resource Handler engine name ``ADB``.
8
+
9
+
The ADB driver is intended for Android VMs launched through minimega's native ``android`` VM type.
10
+
It does not use the QEMU Guest Agent or a QGA socket.
11
+
12
+
Driver Selection
13
+
----------------
14
+
15
+
Android VM Resource Handler process configurations should use:
16
+
17
+
.. code-block:: json
18
+
19
+
{
20
+
"engine": "ADB",
21
+
"adb_serial": "emulator-5554",
22
+
"android_console_port": 5554,
23
+
"android_adb_port": 5555,
24
+
"require_root": true
25
+
}
26
+
27
+
The ``adb_serial`` value is the primary identifier used by the driver. If it is
28
+
not provided, the driver can fall back to ``emulator-<android_console_port>``.
29
+
30
+
ADB, Console Ports, and Serial Names
31
+
------------------------------------
32
+
33
+
Android emulators normally use two sequential ports:
34
+
35
+
* an emulator console port, for example ``5554``;
36
+
* an ADB daemon port, typically ``5555``.
37
+
38
+
The ADB serial uses the console port:
39
+
40
+
.. code-block:: text
41
+
42
+
emulator-5554
43
+
44
+
minimega is authoritative for the actual assigned port values.
45
+
If the requested port pair is unavailable, minimega may assign a different port pair.
46
+
FIREWHEEL should use the actual values reported by minimega before launching the VM Resource Handler.
47
+
48
+
Runtime Paths
49
+
-------------
50
+
51
+
The ADB driver uses Android-native FIREWHEEL runtime paths:
52
+
53
+
.. code-block:: text
54
+
55
+
/data/local/tmp/firewheel/launch
56
+
/data/local/tmp/firewheel/proc
57
+
58
+
Generated call scripts use:
59
+
60
+
.. code-block:: sh
61
+
62
+
#!/system/bin/sh
63
+
64
+
This differs from QGA/Linux VMs, which traditionally use ``/var/launch`` and ``/bin/bash``.
65
+
66
+
Execution Model
67
+
---------------
68
+
69
+
The ADB driver executes VM resources by writing a small runner script under:
70
+
71
+
.. code-block:: text
72
+
73
+
/data/local/tmp/firewheel/proc
74
+
75
+
The runner records:
76
+
77
+
* stdout
78
+
* stderr
79
+
* exit code
80
+
* a started marker
81
+
82
+
This avoids long-lived ADB stream parsing and provides reliable exit-code capture for Android VM resources.
83
+
84
+
Android Shell Limitations
85
+
-------------------------
86
+
87
+
Android guests should not be assumed to provide:
88
+
89
+
* ``/bin/bash``
90
+
* ``/bin/sh``
91
+
* Python
92
+
* systemd
93
+
94
+
VM resources intended for Android should generally use:
95
+
96
+
.. code-block:: sh
97
+
98
+
#!/system/bin/sh
99
+
100
+
Android model components may provide optional Bash or Python runtimes, but these are not required by the ADB driver.
101
+
102
+
Root Access
103
+
-----------
104
+
105
+
The ADB driver can require root by setting ``require_root`` to ``true``.
106
+
This is the default for FIREWHEEL Android experiments that need to configure networking or system-level state.
107
+
108
+
If ``require_root`` is enabled, the Android image must support ``adb root``.
109
+
110
+
If ``require_root`` is false, the driver can connect to non-rooted devices, but VM resources that require privileged operations such as network configuration may fail.
111
+
112
+
File Transfer
113
+
-------------
114
+
115
+
The ADB driver uses ADB sync operations to push and pull files. File transfers are timestamp-filtered where possible using Android ``stat``/``toybox stat``.
116
+
117
+
Reboot Behavior
118
+
---------------
119
+
120
+
The ADB driver can reboot Android guests through ADB.
121
+
Runtime network state such as IP addresses, routes, policy rules, and firewall rules is generally lost on Android reboot.
122
+
Installed APKs typically persist across a normal Android reboot.
123
+
124
+
Android-specific reboot and runtime-state limitations are documented in the Android model component repository.
Copy file name to clipboardExpand all lines: docs/source/system/vm_resource_manager/qga.rst
+8-3Lines changed: 8 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,10 +4,15 @@ QEMU Guest Agent
4
4
================
5
5
6
6
The `QEMU Guest Agent (QGA) <https://wiki.qemu.org/Features/GuestAgent>`_ is a binary file located within a VM which receives QEMU Machine Protocol (QMP) messages and can be used to perform actions within the virtual machine.
7
-
The QGA is the primary way in which the :ref:`vm-resource-handler` communicates with VMs and enables the completion of VM resources.
8
-
Therefore, in order for a VM image to be "FIREWHEEL-compatible" the QGA should be installed on the system and set to run as a service on system start.
7
+
The QGA is the primary way in which the :ref:`vm-resource-handler` communicates with QEMU/KVM VMs and enables the completion of VM resources.
8
+
Therefore, in order for a QEMU/KVM VM image to be "FIREWHEEL-compatible" the QGA should be installed on the system and set to run as a service on system start.
9
9
10
-
When a VM is launched, a virtual serial port that is added which enables the QGA to communicate to FIREWHEEL via the :py:class:`QemuGuestAgentDriver <firewheel.vm_resource_manager.drivers.qemu_guest_agent_driver.QemuGuestAgentDriver>`, which then works with the :ref:`vm-resource-handler` for scheduling and executing VMRs within the VM.
10
+
.. note::
11
+
12
+
Android guests use the ADB driver rather than the QEMU Guest Agent. See
13
+
:ref:`adb-driver`.
14
+
15
+
When a QEMU/KVM VM is launched with QGA support, a virtual serial port that is added which enables the QGA to communicate to FIREWHEEL via the :py:class:`QemuGuestAgentDriver <firewheel.vm_resource_manager.drivers.qemu_guest_agent_driver.QemuGuestAgentDriver>`, which then works with the :ref:`vm-resource-handler` for scheduling and executing VMRs within the VM.
Copy file name to clipboardExpand all lines: docs/source/system/vm_resource_manager/vm_resource_handler.rst
+10-4Lines changed: 10 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,13 +7,19 @@ To facilitate the execution of VMRs, FIREWHEEL uses a :py:class:`VM Resource Han
7
7
In addition to being in charge of loading VMRs on to a VM, it also facilitates executing commands on VMs and extracting information out of the environment (e.g. :ref:`helper_pull_file`).
8
8
9
9
Practically, the :py:class:`VM Resource Handler <firewheel.vm_resource_manager.vm_resource_handler.VMResourceHandler>` dynamically loads in drivers which perform the actual interactions between the VM and the :py:class:`VM Resource Handler <firewheel.vm_resource_manager.vm_resource_handler.VMResourceHandler>`.
10
-
Currently, there is only one driver (:py:class:`QemuGuestAgentDriver <firewheel.vm_resource_manager.drivers.qemu_guest_agent_driver.QemuGuestAgentDriver>`) which handles VMs of type ``QemuVM`` (see :ref:`qemu-guest-agent`).
11
-
However, this dynamic loading function enables future expansion of drivers that will work without changing the :py:class:`VM Resource Handler <firewheel.vm_resource_manager.vm_resource_handler.VMResourceHandler>` [#f1]_.
10
+
FIREWHEEL currently supports multiple VM Resource Handler communication drivers, including:
11
+
12
+
* :py:class:`QemuGuestAgentDriver <firewheel.vm_resource_manager.drivers.qemu_guest_agent_driver.QemuGuestAgentDriver>` for QEMU/KVM VMs using the QEMU Guest Agent. This driver is selected with engine ``QemuVM``. See :ref:`qemu-guest-agent`.
13
+
* :py:class:`ADBDriver <firewheel.vm_resource_manager.drivers.adb_driver.ADBDriver>` for Android guests using the Android Debug Bridge. This driver is selected with engine ``ADB``. See :ref:`adb-driver`.
14
+
15
+
.. note::
16
+
17
+
Additional drivers may be added in the future for other guest communication mechanisms, such as `miniccc <https://www.sandia.gov/minimega/module-28-miniccc-and-the-cc-api/>`_.
18
+
19
+
This dynamic loading mechanism allows FIREWHEEL to support different guest communication mechanisms without changing the VM Resource Handler's scheduling logic.
12
20
13
21
.. _vm-resource-logs:
14
22
15
23
VM Resource Logs
16
24
----------------
17
25
The :py:class:`VM Resource Handler <firewheel.vm_resource_manager.vm_resource_handler.VMResourceHandler>` also facilitates logging VMR data onto the physical host.
18
-
19
-
.. [#f1] For example, we hope to eventually create a driver for `miniccc <https://www.sandia.gov/minimega/module-28-miniccc-and-the-cc-api/>`_.
Copy file name to clipboardExpand all lines: docs/source/system/vm_resource_manager/vmr_in_experiment.rst
+60-8Lines changed: 60 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -101,7 +101,7 @@ These get passed to the program on the command line.
101
101
Finally, the ``vm_resource`` parameter is an optional Boolean that indicates if ``program`` is the name of a script that needs to be loaded on to the VM before execution.
102
102
If ``vm_resource`` is ``True`` then the specified program name is assumed to be the local filename of the file (i.e. not the full path, just the filename) to load on to the VM.
103
103
In this case, like :py:meth:`drop_file <base_objects.VMEndpoint.drop_file>`, the file that corresponds to ``program`` must be part of the ``vm_resources`` list in a model component's ``MANIFEST`` file (see :ref:`vm_resources_field`).
104
-
The user does not need to specify where the file will be dropped on the VM since it will be placed in the appropriate ``/var/launch`` or ``C:\launch`` location (see :ref:`vmr-location`).
104
+
The user does not need to specify where the file will be dropped on the VM since it will be placed in the appropriate launch directory for the VM's communication driver. For QGA Linux guests this is ``/var/launch``; for Windows guests this is ``C:\launch``; for Android guests using the ADB driver this is ``/data/local/tmp/firewheel/launch`` (see :ref:`vmr-location`).
105
105
106
106
Examples
107
107
""""""""
@@ -183,13 +183,22 @@ Another concrete example of how this can be done is shown in the example below w
183
183
184
184
Finally, the :ref:`tests.reboot_mc` has various examples of Python VMRs requesting a reboot.
185
185
186
+
.. note::
187
+
188
+
Guest reboot effects are OS-specific. On Android, runtime network configuration
189
+
such as IP addresses, routes, policy routing rules, and firewall rules is
190
+
generally lost after reboot. Installed APKs typically persist across a normal
191
+
Android reboot. Android resources that need networking after reboot should
192
+
rerun the Android network configuration resources.
193
+
186
194
.. [#] For more details about why this is the case see https://www.howtogeek.com/182817/htg-explains-why-does-windows-want-to-reboot-so-often/
187
195
188
196
.. _vmr-env:
189
197
190
198
VMRs In-Experiment Environment
191
199
------------------------------
192
-
Because of how the VM Resources are executed within the VMs by the :ref:`QEMU Guest Agent <qga-driver>`, users should not make any assumptions about the `environment variables <https://en.wikipedia.org/wiki/Environment_variable>`_ which are available.
200
+
Because VM Resources are executed through a VM Resource Handler driver, users should not make assumptions about the `environment variables <https://en.wikipedia.org/wiki/Environment_variable>`_ available inside the VM.
201
+
The exact environment depends on the guest OS and driver. For example, :ref:`QEMU Guest Agent <qga-driver>` Linux execution and Android ADB execution expose different shell environments.
193
202
That is, users should **NOT** assume that standard Environment variables (e.g. ``$HOME``, ``$USER``, ``$SHELL``, etc.) are available.
194
203
Some software may make assumptions that these common environment variables exist, which may result in odd failures when attempting to run the software.
195
204
Below are some examples of environments which *might* be available.
@@ -211,13 +220,19 @@ Example output from :py:meth:`run_executable <base_objects.VMEndpoint.run_execut
211
220
OLDPWD=/
212
221
_=/usr/bin/printenv
213
222
223
+
Android ADB-based VM resources should generally assume ``/system/bin/sh`` is available, but should **not** assume Bash or Python are installed.
214
224
215
225
.. _vmr-location:
216
226
217
227
Location of VMRs within the VM
218
228
------------------------------
219
229
Prior to the start of :ref:`schedule-negative-time`, VMRs are uploaded onto the VM.
220
-
FIREWHEEL uses the ``/var/launch`` directory on Linux based VMs and the ``C:\launch`` directory on Windows based VMs.
230
+
FIREWHEEL uses driver- and OS-specific launch directories. Common locations include:
231
+
232
+
* ``/var/launch`` for QGA Linux guests
233
+
* ``C:\launch`` for Windows guests
234
+
* ``/data/local/tmp/firewheel/launch`` for Android guests using the ADB driver
235
+
221
236
Inside these directories is a series of directories with :ref:`start times <start-time>` that are used by the scheduled VMRs.
222
237
223
238
For example, if there are VMRs scheduled at -300, -250, -100, and 5 then ``/var/launch`` would look like:
@@ -227,7 +242,7 @@ For example, if there are VMRs scheduled at -300, -250, -100, and 5 then ``/var/
227
242
$ ls /var/launch
228
243
-100 -250 -300 5
229
244
230
-
Inside these ref:`<start-time>` directories is a directory for each VMR which is scheduled at the given time.
245
+
Inside these :ref:`start time <start-time>` directories is a directory for each VMR which is scheduled at the given time.
231
246
232
247
For example, if ``set_hostname.sh`` and ``get_stat.py`` both occurred at ``-250`` it would look like:
233
248
@@ -239,16 +254,17 @@ For example, if ``set_hostname.sh`` and ``get_stat.py`` both occurred at ``-250`
239
254
.. note::
240
255
To access negative time folders you will likely need to use either a full path (e.g. ``/var/launch/-100`` or a specific relative path ``./-100``. This is because the negative sign (i.e. the hyphen) is typically used to express a CLI option for most shell programs.
241
256
242
-
Each of the folders with the VMR name contains a file called ``call_arguments.sh`` which is a dynamically-generated script which executes the VMR.
257
+
Each VMR directory contains a dynamically-generated call-arguments file which executes the VMR.
258
+
On Linux and Android guests this is typically ``call_arguments.sh``; on Windows guests this is typically ``call_arguments.bat``.
243
259
Additionally, if there is other data, scripts, etc. which need to be executed for the given VMR, they are also located in this directory.
244
-
For example, here is the ``/var/launch/-250/set_hostname.sh`` directory:
260
+
For example, on a QGA Linux guest, here is the ``/var/launch/-250/set_hostname.sh`` directory:
245
261
246
262
.. code-block:: bash
247
263
248
264
$ ls /var/launch/-250/set_hostname.sh
249
265
call_arguments.sh set_hostname.sh
250
266
251
-
Here is an example ``call_arguments.sh`` file for ``set_hostname.sh``:
267
+
Here is an example ``call_arguments.sh`` file on a QGA Linux guest for ``set_hostname.sh``:
252
268
253
269
.. code-block:: bash
254
270
:caption: Example ``call_arguments.sh``
@@ -258,14 +274,36 @@ Here is an example ``call_arguments.sh`` file for ``set_hostname.sh``:
Becoming familiar with the locations of VMRs with the VMs is useful for developing and debugging new VMRs.
268
301
302
+
.. note::
303
+
304
+
Android VMRs should generally use ``/system/bin/sh`` and should **not** assume
305
+
``/bin/bash``, ``/bin/sh``, or Python are installed.
306
+
269
307
.. _vmr-output:
270
308
271
309
Extracting VMR data
@@ -300,6 +338,20 @@ This could include running a script that modifies the entire network topology or
300
338
The primary method for scheduling host-based VM resources is through the use of the :py:meth:`run_host_mm_command <base_objects.VMEndpoint.run_host_mm_command>` method, which creates a :py:class:`RunHostExecutableScheduleEntry <base_objects.RunHostExecutableScheduleEntry>` for executing Minimega commands.
301
339
Additional interfaces into host-based scheduling may be added in the future to facilitate common tasks.
302
340
341
+
Host-side VM resources receive several FIREWHEEL-specific environment variables:
342
+
343
+
* ``FIREWHEEL_VM_NAME``
344
+
* ``FIREWHEEL_VM_UUID``
345
+
* ``FIREWHEEL_VM_CONFIG_JSON``
346
+
* ``FIREWHEEL_PYTHON``
347
+
348
+
For Android/ADB-backed VMs, the environment may also include:
349
+
350
+
* ``FIREWHEEL_ADB_SERIAL``
351
+
* ``FIREWHEEL_ANDROID_CONSOLE_PORT``
352
+
353
+
These can be useful for host-side resources that need access to information about the scheduling VM.
The :py:meth:`run_host_mm_command <base_objects.VMEndpoint.run_host_mm_command>` method allows a user to schedule Minimega commands directly on the host.
0 commit comments