path: root/docs/notes.md

# research

## DS / DSi emulators

wanted features:

- **DS and DSi version emulation (DSi for later)**
  
  DS version of pictochat can still receive images with rainbow drawings, but
  doesn't support sending
- **firmware booting**
  
  the phat DS and DS lite's firmware had pictochat built-in instead of as a
  standalone app like on the DSi. to launch pictochat, firmware booting must be
  possible.
- **local multiplayer network emulation**
  
  required for testing sending and receiving
- **local multiplayer packet capture**
  
  required for reverse engineering the packet format
- **local multiplayer packet injection**
  
  required for testing forged packets
- **open source**
  
  DIY packet injection anyone?
- **linux-compatible**
  
  just a preference
- **debugging (ideally using GDB or similar interface)**
  
  if all else fails, reading and/or injecting packets into memory using a
  debugger might be the only option for testing forged packets. a debugger
  could also help with reverse engineering the decoding process of a pictochat
  packet/message

### [DeSmuME][desmume]

- phat ds only
- no firmware booting
- no local multiplayer emulation
+ open source
+ cross-platform
- no debugger

DeSmuME is focused on high quality/performance emulation for single player
titles, and actively rejects requests for WiFi emulation.

### [melonDS][melonds]

+ ds and (experimental) dsi support
+ firmware booting support
+ local multiplayer emulation
+ allows patched-in or real network packet capture using loopback network
  adapter and wireshark
+ loopback network adapter also (probably) allows injecting forged packets
+ open source
+ cross-platform
- no debugger

melonDS is a very new emulator that is still actively being developed. melonDS
is focused on accurate emulation and is the only one on this list with
sufficiently advanced local multiplayer emulation.

### [no$gba][nogba]

+ supports DS and DSi games
+ firmware booting support
- no local multiplayer emulation
- closed source
- windows only
+ very advanced debugger

[toolchaingenericds]: https://bitbucket.org/Coto88/toolchaingenericds/src/master/
[nogba]: https://problemkaputt.de/gba.htm
[melonds]: https://melonds.kuribo64.net/
[desmume]: https://desmume.org/

## MelonDS hacking

source: <https://git.pipeframe.xyz/fork/melonDS>

[masscat-nifi]: <https://web.archive.org/web/20090202194241/http://masscat.afraid.org/ninds/proto_info.php>

- melonDS @ Config > Wifi settings "Local multiplayer features do not use the same network protocols as online play"
- comment @ src/Wifi.cpp:46 "multiplayer host TX sequence"
- references to `RFTransfer_Type{2,3}` @ <https://www.problemkaputt.de/gbatek.htm#dswifirfchip>
- nintendo ds ni-fi protocol @ [masscat-nifi]
- melonDS emulates actual 802.11b frames
- the protocol does not appear to be encrypted:

  ![](../assets/ws-no-encrypt.png)

  the string `lork` is visible as plain text in the hexdump (offset 0x0056),
  which appears to be some kind of 16-bit encoding of the username set on the
  emulator used to capture these packets (this is likely UCS-2 and not UTF-16
  as suggested by [masscat-nifi] which specifies UCS-2).
- WEP is not used for local multiplayer. This was verified after using
  Wireshark to dissassemble the IEEE802.11 header present in all captured
  frames. None of the captured frames had the WEP encrypt flag set.
- The messages are not sent as single packets. The Ni-Fi protocol appears to set
  up a constant stream, and messages are sent across multiple frames.
- PictoChat does not appear to send messages when you are in a chat room by
  yourself, so local multiplayer emulation is required for capturing message
  content

### Message sizing/cropping

- PictoChat automatically crops messages to the smallest height (at the fixed
  intervals shown as notebook lines in the edit field on the bottom screen).
  Message content can be on the notebook line itself without causing the
  cropping algorithm to 'allocate' more space; the allocation only happens if
  any of the pixels on the line *below* the colored line are used. The method
  used to crop the messages also ensures that the username label in the top
  left does not obstruct or remove any content.

  ![](../assets/pictochat-msg-height-1-4.png)
  ![](../assets/pictochat-msg-height-5-draw.png)
- In addition to being cropped before sending, the message border appears to
  mask the drawing:

  ![](../assets/pictochat-msg-crop.png)
  ![](../assets/pictochat-msg-crop-lork2.png)

  The above image was obtained after the following steps:
  - Fill in the top row of the message with the black pen, ensuring the entire
    notebook line is colored in as well
  - Send message (displayed as top message)
  - Copy message
  - Erase small portion of black area on the right side (displayed as bottom
    message)
  
  Notable observations:
  - The message content has a 1 pixel border (padding/margin) on all sides
    (except the bottom and corners) that can not be filled in. The bottom
    border is correctly displayed for received messages, but the edit field has
    an additional row of pixels that directly touch the message border on the
    bottom which is not visible in the received version of the message (i.e. is
    being masked).
  - When any message is cloned, additional pixels that were masked on the top
    screen become visible again in the edit field. This includes the bottom row
    of pixels, as well as the two rows of pixels shown in the single line
    message picture.
  - The drawable area (including obstructed top-left corner) is 228x80 pixels,
    while the entire message is actually 256x80 pixels:

    ![](../assets/draw-area.svg)

    The above figure illustrates the area which is 'drawable' (i.e. the area in
    which the pen tool works). Fiddled messages that go outside this area are
    received normally, but masked slightly differently when re-sent (chunk left
    of username label appears to be masked). The actual image data is
    unmodified even after being forwarded by normal systems (non-fiddling
    systems).

Observed package counts (no resends) for messages of different sizes:
|draw area|display height|packet#|
|-|-|-|
|top-left pixel of row 1 only|1|13|
|row 1|1|13|
|row 2|2|25|
|row 3|2|25|
|rows 1-2|2|25|
|rows 1-3|3|38|
|rows 4-5|3|38|
|rows 2-3|3|38|
|rows 1-4|4|51|
|rows 1-5|5|64|
|rows 2-5|5|64|

Notable:
- Messages that draw below line 1 while keeping line 1 empty can't be cropped
  to utilize line 1 as the username label is in the way. The captured packet
  counts suggest that the content of line 1 is still sent (although empty).
- The messages are cropped *before* sending, as suggested by the packet counts
- The message content is masked when displayed in the message log on the top
  screen

### Message content

![](../assets/pictochat-msg-pattern.png)
![](../assets/ws-msg-pattern.png)

- All messages with interesting content have Ni-Fi header type 1 (CMD).
- PictoChat messages appear to be sent over frames of length 0xf6 (246)
  regardless of actual message size.
- All frames appear to be sent exactly 5 times. 'New' frames are indicated with
  a 0/2 boolean (0=resend, 2=new) at various points in the packet data. This
  resend field may not apply to non-pictochat packets, and [GBATEK][gbatek]
  describes this field completely differently. PictoChat resend packets can
  safely be dropped without impacting communication (while cutting the packet
  dump file size down by ~80%).
- The message content appears to be sent unencrypted (patterns of 0x0 and 0x1
  nibbles clearly visible in hexdump).
- Offset 0x004d - 0x00ec appears to be used to send the actual message content:

  ![](../assets/pictochat-msg-crop-lork2.png)
  ![](../assets/ws-msg-fill-top.png)
  ![](../assets/ws-msg-fill-mid.png)
  ![](../assets/ws-msg-fill-bot.png)

  (excerpts from the start, middle and end frames of a completely filled
  message)
- The completely filled message also shows an interesting pattern at the end,
  hinting at a slightly odd image codec
- Message length **can not** be used to consistently check which packets
  contain PictoChat message data.

<!-- #### MAC addresses

The following table of addresses may be visible in the packet captures.

|address|label|source|
|-|-|-|
|`00:09:bf:11:22:33`|Default firmware MAC|[melonDS/src/SPI\_Firmware.h:37](../melonDS/src/SPI_Firmware.h)|
|`03:09:bf:00:00:00`|Multiplayer CMD|[melonDS/src/Wifi.cpp:42](../melonDS/src/Wifi.cpp)|
|`03:09:bf:00:00:10`|Multiplayer Reply|[melonDS/src/Wifi.cpp:43](../melonDS/src/Wifi.cpp)|
|`03:09:bf:00:00:03`|Multiplayer ACK|[melonDS/src/Wifi.cpp:44](../melonDS/src/Wifi.cpp)|
|`00:f0:77:77:77:77`|Access point|[melonDS/src/WifiAP.cpp:37](../melonDS/src/WifiAP.cpp)|
|`10:00:de:ad:be:ef`|Instance 1 (`lork`) firmware MAC|me|
|`20:00:de:ae:02:ff`|Instance 2 (`lork2`) firmware MAC|me|
|`ff:ff:ff:ff:ff:ff`|Broadcast||

In melonDS's settings, the firmware MAC addresses for `lork` and `lork2` are
set to `11:00:de:ad:be:ef` and `22:00:de:ad:be:ef` respectively, but these
always get mangled and sent as the addresses noted in the table above. I have
no idea why this happens. -->

#### Analysis

See [the wireshark dissector](../wireshark/pictochat.lua)

<!-- ![](../assets/ws-msg-fill-mid.png)

|offset|type|description|
|-|-|-|
|0x00|`u32`|Ni-Fi: Magic (0x4e494649)|
|0x04|`u32`|Ni-Fi: SenderID (melonDS InstanceID)|
|0x08|`u32`|Ni-Fi: Type|
|0x0c|`u32`|Ni-Fi: Length (after Ni-Fi header)|
|0x10|`u64`|Ni-Fi: Timestamp|
|
|0x18|`u16`|PictoChat: Message type??? (0)|
|0x1a|`u16`|PictoChat: Resend (2=New, 0=Resend)|< !-- I assume u16 because the next byte is always 0x00 -- >
|0x22|`u16`|PictoChat: Length (offset 36 of complete packet length)|
|0x28|`u8[6]`|PictoChat: multiplayer CMD MAC (melonDS Wifi::MPCmdMAC)|
|0x2e|`u8[6]`|PictoChat: sender MAC|
|0x34|`u8[6]`|PictoChat: sender MAC (again)|
|0x3a|`u16`|Unknown: counter|
|0x4a|`u16`|PictoChat: Message content offset|
|0x4e|`u8[0xa0]`|PictoChat: Message data (in 8x8 tiles where each byte represents two pixels as nibbles)|
|0xee|`u16`|PictoChat: packet sequence number|
|0xf0|`u16`|PictoChat: copy of 0x1a (Resend)|
|0xf2|`u32`|Unknown: constant 0x93ffb8b6|

TODO: 0x4e(PictoChat msg data offset) - 0x18(Ni-Fi header length) == 0x36
-->

### Fiddling

#### Corrupt

After editing melonDS source code to corrupt the local multiplayer message
buffer in known content locations before sending the message to the FIFO
buffer:

![](../assets/pictochat-msg-corrupt-lork.png)
![](../assets/pictochat-msg-corrupt-lork2.png)

This shows a few important details:

- Messages are not limited to the apparent 228x80 size, and lines are visible
  outside the message border (there is also message content behind the username
  label in the above screenshots, but the username label is displayed above the
  message)
- The ordering of pixels in the messages is not reading order
- Message content is not checked or validated in any way

#### Fill

![](../assets/pictochat-fiddle-fill-lork.png)
![](../assets/pictochat-fiddle-fill-lork2.png)

After dissecting the messages, the bounds of the actual message content, as
well as how to consistently differentiate between pictochat messages and other
messages were used to create a fiddle filter that fills an entire message with
palette index 3 (hot pink). This clearly shows the actual bounds of the
pictochat message.

Cloning a filled message results in an interesting visual glitch. The message
content appears to be duplicated, as it does not actually contain enough pixel
data to fill almost the entire bottom screen.

![](../assets/pictochat-fiddle-fill-glitch.png)

## Room host

The system that initially joins an empty room appears to become the room host.
This system ends up never sending messages of Ni-Fi type 2 (Reply), while the
packets from other systems after joining an existing room are sent as Ni-Fi
type 2 exclusively.

![](../assets/ws-announce-vs.png)

(Bottom Wireshark window is the room host, top window shows type != 2 Ni-Fi
messages from the system that joined later)

## Drawing format

- The drawings appear to be split into 8x8 sprite tiles. TODO: pixel order
  within tiles + TODO: sprite ordering (likely both reading order).

### Palette

|idx|hex|
|-|-|
|0|transparent (white)|
|1|`#000000` <b style="color: #000000">■</b>|
|2|`#d3cbc3` <b style="color: #d3cbc3">■</b>|
|3|`#eb00eb` <b style="color: #eb00eb">■</b>|
|4|`#fb008a` <b style="color: #fb008a">■</b>|
|5|`#fb0028` <b style="color: #fb0028">■</b>|
|6|`#fb4900` <b style="color: #fb4900">■</b>|
|7|`#fba200` <b style="color: #fba200">■</b>|
|8|`#e3f300` <b style="color: #e3f300">■</b>|
|9|`#82fb00` <b style="color: #82fb00">■</b>|
|a|`#10fb20` <b style="color: #10fb20">■</b>|
|b|`#00fbba` <b style="color: #00fbba">■</b>|
|c|`#00c3fb` <b style="color: #00c3fb">■</b>|
|d|`#0079fb` <b style="color: #0079fb">■</b>|
|e|`#0030fb` <b style="color: #0030fb">■</b>|
|f|`#2800fb` <b style="color: #2800fb">■</b>|

## More interesting sources / link dump

[gbatek]: https://www.problemkaputt.de/gbatek.htm
- <https://www.problemkaputt.de/gbatek.htm#dswifiieee80211frames>
- <https://www.problemkaputt.de/gbatek.htm#dswifinintendobeacons>

## Reassembly

The pictochat protocol itself also seems to be made up of several layers, one
of which splits large messages over multiple frames. After writing more lua
code to reassemble these chunks, the following observations were made:

- There is a fixed 0x24 (36) byte header before all drawings
- The size of the reassembled messages for pictochat drawings exactly follows
  the equation $f(x) = 36 = 2048x$ where $x$ is the number of rows the message
  shows up as. The row count of the message does not appear to be stored
  anywhere.
- The first 8 bytes of reassembled messages (drawings, room join) consists of a
  constant 0x03, a message type indicator (1 byte) and the address of the
  author/subject (6 bytes).
- The remaining 28 bytes of the reassembled message header are constant for
  drawings. These bytes appear to be some kind of padding as changing them
  appears to have no effect and does not show up in the drawing.

Using the reassembled message data, the following message was decoded as an
experiment:

![](../assets/first-decode-pictochat.png)

Wireshark allows you to easily save the binary content of a dissection field to
a file:

![](../assets/first-decode-ws.png)

Resulting decoded image:

<div style="background-color: black; padding: 4px; font-size: 0px; display: inline-block;">
<img src="../assets/first-decode.png"/>
</div>

## Unsure/notes

- Is the endianness of the DS properly emulated?
- Where is a message's destination (pictochat room) defined?

## TODO:

- are message-resends required or can the packets be dropped?
- user identifier / login / announcement procedure?
- actual message format
    * resolution
    * pixel ordering
    * palette color indices (pixels are 1 nibble)
- what types of pictochat packets are there? (i.e. how are room join/leave
  events broadcast?)