Multimodal LLM based Gemini Model for ASR

On-Device Federated Learning of Large Models

LocLok: Protecting Location Privacy in Smartphones

DPCube: Releasing Differentially Private Data Cubes for Health Information

Multimodal LLM based Gemini Model for ASR

I work on multimodal Gemini model for ASR (Automatic Speech Recognition) and improved ASR result at Google on the Gemini 1.5 architecture by supervised fine-tuning the pre-trained model with various new algorithms designed for multimodal LLM. Due to confidentiality I can only say it's the best Gemini ASR result as of July 2024.

To the top.

On-Device Federated Learning of Large Models

I led the full-Conformer based ASR model training under federated learning. By leveraging the real data from real users, the on-device federated trained model can improve the ASR performance. However, there are several constraints on devices including the RAM size, CPU computation power and network bandwidth etc. I was the first person at Google to achieve the federated learning of large Conformer model, and achieved best ASR model under federated learning.

To the top.

LocLok: Protecting Location Privacy in Smartphones

Concerns on location privacy frequently arise with the rapid development of GPS enabled devices and location-based applications. However, there is no offically privacy-preserving cell phone apps which can protect users' location privacy while enabling the GPS based services. In academia, the popular method is to replace a position of latitude and longitude with a randomly generated area, called spatial cloaking. For more details, please go to http://forum.loclok.com.

Click here to view a demo system.

Our technique can rigorously protect location privacy even when the attacker is acquainted with the user. For example, the attacker knows exactly the moving habit of the user and the historically visited places of the user. Our technique can still prove the current location privacy is protected by the state-of-art differential privacy. Furthermore, we prove that our technique is the optimal solution to satisfy the guarantee of differential privacy.

I show an example as follows. Figure 1 shows a real trajectory on a real map where the two axes represent longitude and latitude. A user travaled in 500 timestamps. The corresponding grid map of the same trajectory is shown in Figure 2. Then Figure 3 shows the released trajecotry of existing method; while Figure 4 is the released trajectory of our method. Obviously our method provides more utility, which is also verified in Figure 5 where the distances between the true locations and released locations are demonstrated. LM is existing method of Laplace Mechanism; PIM is our method of Planar Isotropic Mechanism. The formal proof and experiment settings can be found in our paper.

To the top.

DPCube: Releasing Differentially Private Data Cubes for Health Information

As we all know that health data is highly private. For example, a patient's disease should never be exposed to anyone without the patient's consent. On the other hand, health data is very useful for clinical treatment improvement, or medical research institute. Then the question is how to release health data so that it can protect the sensitive part while releasing the useful part. Or generally, if we have a database containing both private and useful information, how to use such data without breaching any privacy?

DPCube is a practical and rigorous method to tackle this. It satisfies the state-of-art differential privacy guarantee for privacy preservation. On the other hand, it provides useful information of the data. The brief components of DPCube is desribed as follows.

We also have the Matlab code to implement DPCube, which is available upon request. Following figure shows the interface of a data releasing scenario.

To the top.

FedAQT: Accurate Quantized Training with Federated Learning

This is our FedAQT work under federated learning.

Federated Pruning: Improving Neural Network Efficiency with Federated Learning

This is our FedPruning work under federated learning.

An Adversarial Machine Learning Model

ML model is known for it's vulnerability under adversarial attacks. Even for the state-of-art ML models, it can be easily fooled by adding some amount of noise.

In the following example, we shows that a cat (on the left) in tiny ImageNet can be misclassfied to other objects when adding little noises which are not even perceived by humans. Although the three images on the right side look similar to the original cat, they are classified as sea cucumber, desk and dragonfly. Note that the ML model is the ResNet18, one of the best vision models as of 2018 with about 76% accuracy.

The attacking time to generate one adversarial sample is about 60~90 seconds. Our code uses the Carlini algorithm, and is available upon request. Also note that the perturbation can also be less than 1.0 pixel (in the 0~255 image scale) under the attack.

To the top.

A P2P Video Sharing System

Many of us watch online videos everyday, either on Youtube or other video providers. However, the technique behind the video sharing is not hard. Here I show a simple video sharing system with the original code.

In my system, there are three major parts, a tracker, a super peer and many peers. The tracker is a MySQL database service. It contains the all the information about the availabe videos. If a user, which is a peer, wants to browse the videos of our system, it sends a query request to the tracker, which returns all the channels to the peer. Each of the channel has a super peer, containing the detailed information of the channel. If a user selected a channel, then it will build a TCP connection with the super peer. Then the super peer accepts the TCP connection and add the user to the neighbors of audiences.

In the TCP connection, the video content is transformed in the unit of data packets. As shown in the following figure, a peer has a data buffer area to contain these packets.

The classes used in this program is summarized as follows.

To download the original code written in C++, please click the following: super peer, tracker01, tracker01.client and tracker02.

To the top.

Deep Learning Certificate

Machine Learning Certificate

I-Corps Innovation Certificate

Deep Learning Certificate

I obtained the Deep Learning certificate from deeplearning.ai.

To the top.

Machine Learning Certificate

I obtained the Machine Learning certificate from Stanford University.

To the top.

I-Corps Innovation Program

I am very lucky to participate the I-Corps program supported by National Science Foundation in 2016. Although my startup did not work after exploring the business opportunities, I am grateful to the I-Corps program, shown in the following image.

To the top.

Awards

• Amazon graduate research symposium, 2017
• IEEE S&P student PC travel award, 2017
• NSF I-Corps award as entrepreneur lead, 2016
• CCS travel award, 2015.
• NSF ICDE scholarship, 2012.
• Ph.D. Fellowship, Laney Graduate School of Emory University, 2011-Present
• Scholarship of Foxconn at Tsinghua University, Beijing, China 2009.