Projects | Zhiyuan Liu

A tool to make paper more colorful

Wed, 27 Apr 2022 12:51:44 +0000

论文个性化系统

功能

主页导航，跳转功能页面。
自定义论文风格。功能：上传风格文件夹，生成自定义的论文风格。接口：/style/custom
上传文件，获得风格化论文。接口：style/file
上传文件夹，获得风格化论文系列文件。style/file_folder

运行

安装requirements.txt 里面的库。在项目目录路径，运行即可。

 pip install -r requirements.txt

由于pypdf2库，内部代码逻辑原因，需要对其进行一定改动。找到pypdf2/pdf.py 文件。跳转第1635行，注释掉（如下操作）。

 python index.py

浏览器上访问 http://127.0.0.1:5000/ 即可。

小提示

进入http://127.0.0.1:5000/ （主页）后，通过主页导航，进入上传文件页面，选择风格，点击上传便能得到风格化的论文pdf（参考如下Figure 2图）。
由于github上传文件大小限制，只为用户准备了风景图。用户通过自定义论文风格，添加更多专属style。
上传的文件或者文件夹会保存在uploads文件夹中，返回的风格化文件会保存在handle_pdf文件夹中。

效果展示

Figure 1：效果对比图（上：原论文pdf，下：风格化的pdf）

Figure 2：上传页面效果图

Figure 3：主页

Crawl review data and analyze

Wed, 27 Apr 2022 12:00:54 +0000

数据特征工程实战（简易版）

概览

该项目较为简单，新手需配好python环境。作者使用 python进行开发，jupyter notebook 作为编辑器。
学习该项目您能对爬虫、数据分析与数据可视化等知识获得初步了解。
result_images是存储可视化图片的文件夹

运行

先安装python包

 pip install -r requirements.txt

两种方式运行：

运行py文件

 python main.py

或使用jupyter notebook

下载Anaconda，将jupyter notebook调出，打开main.ipynb文件，运行即可。

实现过程

数据获取：从豆瓣《飞驰人生》评论页获取数据
数据清洗：清洗掉获取数据中的脏数据
数据处理：将有价值的数据提取出来，使用numpy及pandas库进行分析建模
具体详情，转至我的博客

效果展示

Figure 1：词云图

Figure 2：电影评级图

Figure 3：评论数量图

Figure 4：评论均值图

Shanghai taxi's od data analysis

Thu, 15 Jul 2021 00:00:00 +0000

shanghai_spatial_data_analysis

Introduction

A standard procedure to deal with the spatial data (the taxi data of shanghai). We get the Shanghai’s taxi data from 上海市出租车数据集 (a open source dataset).

In this project, you will learn:

How to carry on data analysis.
How to handle the data and dive out the intriguing feature which may be useful for business strategy and optimization.
How to show the spatial data in the map intuitive.

The Prior knowledge that you should know:

Data analysis tool: numpy, pandas, geopandas, shapely.
Be patient, positive and keep going.

Run

To run the code efficiently on your computer, you shall do like below:

install the corresponding library.

pip install -r requirements.txt

If you can not install successfully, that must be the version conflict. So try to read this video geopandas的安装.

Download the 上海市出租车数据集 and put into ./data-sample/taxi_sh/*.

Then, begin to learn according to the number’s order of ipynb files. So let’s begin from the shanghai_data_analysis_1.ipynb.

Figure show

1.Taxi's numbers for each hour

2.The position distribution of taxi in the early morning

3.The gif for postion distribution in each hour

4.Start to end distribution

5.Heat map for position distribution of taxi

Predict repeat buy on e-commerce platforms

Sat, 27 Apr 2019 00:00:00 +0000

Introduction

A project to predict the repeat buy rate for Tianmao. The task is open for everyone in Tianci platform. Although, it’s a learning contest, it does not mean simple. In this project, we are ranked 132th among 7,041 teams which obtained 0.6871 for metric AUC.

Dataset can be download on 阿里云天池之天猫复购率预测

datasets position and structure

Overview

We divided the predict task into four parts: 1. exploring data 2. feature engineering 3. create model 4. model optimization

Exploring data: In this part, we just use some basic usage of pandas and plot some intuitive graphs to make us more familiar with the data. Actually, we do nothing operation to the origin data during the time.
Feature engineering: If we want to predict whether someone will buy the same merchant’s product, it’s crucial to dive into the behavior of users and merchants. After we know the behavior pattern of users and merchant, we shall dig into the connection of the two objects which means we should covert these infomation to numerical data in the field of data analysis.
Create model: Just like a combination punch from WTF pro boxers, we use the Logistic, SVM, KNN, Desition Tree, Naive bayes, Bagging of DT(RandomForest), Boosting of DT(GDBT,XGBoost) for predition tasks in the machine learning. In this section, we build all kinds of model like above and judge which one is the best.
Model optimization: Owing to the imblance of position/negative samples, we make three optimization schemes: Ensemble model, Smote over-sampling and Neural network.

Before learning

What you should knew ?

Data analysis tool: numpy, pandas.
Machine learning tool like sklearn and principle of basic machine learning algorithms.

Install library before running:

pip install -r requirements.txt

If you cannot install some libraries like imblearn or torch, it doesn’s matter. Just go ahead, we use them in the part four (Model optimization).

Projects | Zhiyuan Liu

A tool to make paper more colorful

论文个性化系统

功能

运行

效果展示

Crawl review data and analyze

数据特征工程实战（简易版）

概览

运行

实现过程

效果展示

Shanghai taxi's od data analysis

shanghai_spatial_data_analysis

Introduction

Run

Figure show

Predict repeat buy on e-commerce platforms

Introduction

Overview

Before learning

Some figures