中文说明请点这里

• Python
• Opencv3
• Tensorflow

• Adb tools
• Android Phone

• iPhone
• Mac
• WebDriverAgent
• facebook-wda
• imobiledevice

• Multiscale search
• Fast search
• CNN-based coarse-to-fine model

For algorithm details, please go to https://zhuanlan.zhihu.com/p/32636329.

Notice: CV based fast-search only support Android for now

Before running our code, connect to your phone via USB.

If Android phone, open the USB debugging at developer options enter adb devices to ensure that the list is not empty. If iPhone, please ensure that you have a mac. Then following this link for preparation.

It is recommended to download the pre-trained model following the link below and run the following code

python nn_play.py --phone Android --sensitivity 2.045

You can also try play.py by running the following code

python play.py --phone Android --sensitivity 2.045

• --phone has two options: Android or IOS.
• --sensitivity is the constant parameter that controls the pressing time.
• nn_play.py uses CNN-based coarse-to-fine model, supporting Android and IOS (more robust)
• play.py uses multiscale search and fast search algorithms, supporting Android and IOS (it may fail sometimes in other phones)

Our method can correctly detect the positions of the man (green dot) and the destination (red dot).

It is easy to reach the state of art as long as you like. But I choose to go die after 859 jumps for about 1.5 hours.

Here is a video demo. Excited!

CNN train log and train&validation data avaliable at

• Baidu Drive
• Google Drive

Training: download and untar data into any directory, and then modify self.data_dir in those files under cnn_coarse_to_fine/data_provider directory.

Inference: download and unzip train log dirs(train_logs_coarse and train_logs_fine) into resource directory.

0. Download and untar data into any directory, and then modify self.data_dir in those files under cnn_coarse_to_fine/data_provider directory.
1. base.large is model dir for coarse model, base.fine is model dir for fine model, other dirs under cnn_coarse_to_fine/config are models we don't use, but if you have interests, you can try train other models by yourself.
2. Run python3 train.py -g 0 to train your model, -g to specify GPU to use, if you don't have GPU, training model is not recommended because training speed with CPU is very slow.
3. After training, move or copy .ckpt file to train log dirs(train_logs_coarse and train_logs_fine) for use.