Call/WhatsApp/Text: +44 20 3289 5183

Question: Implementing a big data solution

22 May 2024,6:08 AM

Implement a big data solution (i.e. you must use the tools taught in this course, pySpark DataFrames, RDDs or pySpark SQL, or AWS services presented in the course or a combination thereof). Note that solutions based on pandas, excel or similar non parallel approaches will NOT be awarded any implementation marks.

1. Programmatically confirm that all papers have unique IDs and output the number of papers in the file.

2. What is the average number of authors per paper?

3. How many different journals were the papers published in?

4. Find the 5 authors with the highest number of publications. Give their names along with the number of publications they contributed to.

5. To gain some additional information about publication quality, you’d like to join the paper information with the journal information you have. Following this, find the top 5 authors with the highest cummulative impact factor (notice that journals have different impact factors listed in the journal file in the IF column). Output both the author information and the cummulative impact factor.

6. You’d like some additional information about publication trends. How many publications with impact factor > 1 were published in each of the years between 2010-2020? Ensure that your answer for each year is visible in your report.

 

You will be using two datasets in this assignment. The first is a dataset of academic publications, a subset of the S2ORC dataset (https://github.com/allenai/s2orc). The second dataset contains some journal information. This file, journal information.csv is available in CSV format from Blackboard. The file has a header, and each non empty row contains information about a single journal. The full filepath on Databricks for this file should be: /FileStore/tables/journal information.cs

 

You should write a 3,000 word structured report that presents and explains your solutions, justifies your choice of techniques and discusses the implications of your answers for your client along with exploring any further investigations.

Expert answer

This Question Hasn’t Been Answered Yet! Do You Want an Accurate, Detailed, and Original Model Answer for This Question?

 

Ask an expert

 

Stuck Looking For A Model Original Answer To This Or Any Other
Question?

Related Questions

As a financial analyst for Muffin Construction

As a financial analyst for Muffin Construction, you have been asked to recommend the method of financing the acquisition of new equipment needed by the firm. The equipment has a useful life of eight years. If purchased, the equipment, which costs $700,000, will be depreciated under MACRS rules for 7-year class assets. If purchased, the needed funds can be borrowed at a 10 percent pretax annual rate. Muffin’s weighted after-tax rate of capital is 12 percent. The actual salvage value at the end of eight years is expected to be $50,000. Muffin’s marginal ordinary tax rate is 40 percent. Annual, beginning-of-year ...
Assume in problem 15-1 that Minetellos tax rate is 40%

Assume in problem 15-1 that Minetellos tax rate is 40% and that the equipments CCA would be $20,000 per year. If the company leased the asset on a 2-year lease, the payment would be $22,000 at the beginning of each year. If Minetello borrowed and bought, the bank would charge 10% interest on the loan. In either case, the equipment is worth nothing after 2 years and will be discarded. Should Minetello lease or buy the equipment? DRAFT/STUDY TIPS

The bus costs $1.8 million, with a CCA rate of 25 percent.

Nash Business School is considering whether to lease or buy a shuttle bus for students travelling between its two campuses. The bus costs $1.8 million, with a CCA rate of 25 percent. The lease payments are $400,000 per year for the next five years, with payments made at the beginning of each year. There is no maintenance cost, and the expected market value of the bus after five years is zero. Assuming the lease qualifies as an operating lease and the cost of borrowing is 5 percent should Nash Business School lease the bus or buys it? Universities do not pay tax. DRAFT/STUDY ...
Lester's Gym wants to expand its facilities by obtaining $62,000

Lester's Gym wants to expand its facilities by obtaining $62,000 worth of new exercise equipment. The equipment will have a 4-year life, belongs in a 25% CCA class, and will have no residual value. The cost of borrowed funds is 9.5 percent, the weighted average cost of capital is 12% and the tax rate is 40 percent. Assume the lease payments are payable at the beginning of each year and the lease qualifies as a true tax lease. What is Lester’s breakeven lease payment? DRAFT/STUDY TIPS

What Clients Say About Us

Copyright © 2012 - 2024 Apaxresearchers - All Rights Reserved.

WhatsApp us